Compare commits

..

No commits in common. "master" and "v4.5.2" have entirely different histories.

296 changed files with 3865 additions and 30328 deletions

View File

@ -22,10 +22,6 @@ jobs:
java-version: '11' java-version: '11'
distribution: 'adopt' distribution: 'adopt'
- name: Load custom deps
run: |
mvn install -DskipTests=true --file pom.xml
- name: Run all tests - name: Run all tests
run: | run: |
mvn -B clean test -DskipTests=false --file pom.xml mvn -B clean test -DskipTests=false --file pom.xml

View File

@ -1,56 +1,12 @@
# Qortal Project - Qortal Core - Primary Repository # Qortal Project - Official Repo
The Qortal Core is the blockchain and node component of the overall project. It contains the primary API, and ability to make calls to create transactions, and interact with the Qortal Blockchain Network.
In order to run the Qortal Core, a machine with java 11+ installed is required. Minimum RAM specs will vary depending on settings, but as low as 4GB of RAM should be acceptable in most scenarios. ## Build / run
Qortal is a complete infrastructure platform with a blockchain backend, it is capable of indefinite web and application hosting with no continual fees, replacement of DNS and centralized name systems and communications systems, and is the foundation of the next generation digital infrastructure of the world. Qortal is unique in nearly every way, and was written from scratch to address as many concerns from both the existing 'blockchain space' and the 'typical internet' as possible, while maintaining a system that is easy to use and able to run on 'any' computer.
Qortal contains extensive functionality geared toward complete decentralization of the digital world. Removal of 'middlemen' of any kind from all transactions, and ability to publish websites and applications that require no continual fees, on a name that is truly owned by the account that registered it, or purchased it from another. A single name on Qortal is capable of being both a namespace and a 'username'. That single name can have an application, website, public and private data, communications, authentication, the namespace itself and more, which can subsequently be sold to anyone else without the need to change any type of 'hosting' or DNS entries that do not exist, email that doesn't exist, etc. Maintaining the same functionality as those replaced features of web 2.0.
Over time Qortal has progressed into a fully featured environment catering to any and all types of people and organizations, and will continue to advance as time goes on. Brining more features, capability, device support, and availale replacements for web2.0. Ultimately building a new, completely user-controlled digital world without limits.
Qortal has no owner, no company on top of it, and is completely community built, run, and funded. A community-established and run group of developers known as the 'dev-group' or Qortal Development Group, make group_approval based decisions for the project's future. If you are a developer interested in assisting with the project, you meay reach out to the Qortal Development Group in any of the available Qortal community locations. Either on the Qortal network itself, or on one of the temporary centralized social media locations.
Building the future one block at a time. Welcome to Qortal.
# Building the Qortal Core from Source
## Build / Run
- Requires Java 11. OpenJDK 11 recommended over Java SE. - Requires Java 11. OpenJDK 11 recommended over Java SE.
- Install Maven - Install Maven
- Use Maven to fetch dependencies and build: `mvn clean package` - Use Maven to fetch dependencies and build: `mvn clean package`
- Update Maven dependencies: `mvn install`
- Built JAR should be something like `target/qortal-1.0.jar` - Built JAR should be something like `target/qortal-1.0.jar`
- Create basic *settings.json* file: `echo '{}' > settings.json` - Create basic *settings.json* file: `echo '{}' > settings.json`
- Run JAR in same working directory as *settings.json*: `java -jar target/qortal-1.0.jar` - Run JAR in same working directory as *settings.json*: `java -jar target/qortal-1.0.jar`
- Wrap in shell script, add JVM flags, redirection, backgrounding, etc. as necessary. - Wrap in shell script, add JVM flags, redirection, backgrounding, etc. as necessary.
- Or use supplied example shell script: *start.sh* - Or use supplied example shell script: *start.sh*
## IntelliJ IDEA Configuration
- Run -> Edit Configurations
- Add New Application
- Name: qortal
- SDK: java 11
- Main Class: org.qortal.controller.Controller
- Program arguments: settings.json -Dlog4j.configurationFile=log4j2.properties -ea
- Environment variables: Djava.net.preferIPv4Stack=false
# Using a pre-built Qortal 'jar' binary
If you prefer to utilize a released version of Qortal, you may do so by downloading one of the available releases from the releases page, that are also linked on https://qortal.org and https://qortal.dev.
# Learning Q-App Development
https://qortal.dev contains dev documentation for building JS/React (and other client-side languages) applications or 'Q-Apps' on Qortal. Q-Apps are published on Registered Qortal Names, and aside from a single Name Registration fee, and a fraction of QORT for a publish transaction, require zero continual costs. These applications get more redundant with each new access from a new Qortal Node, making your application faster for the next user to download, and stronger as time goes on. Q-Apps live indefinitely in the history of the blockchain-secured Qortal Data Network (QDN).
# How to learn more
If the project interests you, you may learn more from the various web2 and QDN based websites focused on introductory information.
https://qortal.org - primary internet presence
https://qortal.dev - secondary and development focused website with links to many new developments and documentation
https://wiki.qortal.org - community built and managed wiki with detailed information regarding the project
links to telegram and discord communities are at the top of https://qortal.org as well.

View File

@ -1,4 +1,3 @@
{ {
"apiDocumentationEnabled": true, "apiDocumentationEnabled": true
"apiWhitelistEnabled": false
} }

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 160 KiB

File diff suppressed because it is too large Load Diff

View File

@ -2,9 +2,7 @@
## Prerequisites ## Prerequisites
* AdvancedInstaller v19.4 or better, and enterprise licence. * AdvancedInstaller v19.4 or better, and enterprise licence if translations are required
* Qortal has an open source license, however it currently (as of December 2024) only supports up to version 19. (We may need to reach out to Advanced Installer again to obtain a new license at some point, if needed.
* Reach out to @crowetic for links to the installer install files, and license.
* Installed AdoptOpenJDK v17 64bit, full JDK *not* JRE * Installed AdoptOpenJDK v17 64bit, full JDK *not* JRE
## General build instructions ## General build instructions
@ -12,12 +10,6 @@
If this is your first time opening the `qortal.aip` file then you might need to adjust If this is your first time opening the `qortal.aip` file then you might need to adjust
configured paths, or create a dummy `D:` drive with the expected layout. configured paths, or create a dummy `D:` drive with the expected layout.
Opening the aip file from within a clone of the qortal repo also works, if you have a separate windows machine setup to do the build.
You May need to change the location of the 'jre64' files inside Advanced Installer, if it is set to a path that your build machine doesn't have.
The Java Memory Arguments can be set manually, but as of December 2024 they have been reset back to system defaults. This should include G1GC Garbage Collector.
Typical build procedure: Typical build procedure:
* Place the `qortal.jar` file in `Install-Files\` * Place the `qortal.jar` file in `Install-Files\`

View File

@ -1,5 +0,0 @@
This is the production hsqldb-2.7.4 with the manifest file updated
Sealed: false
Allows the addition of the custom Qortal HSQLDBPool and Monitoring Classes

69
pom.xml
View File

@ -3,13 +3,12 @@
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<groupId>org.qortal</groupId> <groupId>org.qortal</groupId>
<artifactId>qortal</artifactId> <artifactId>qortal</artifactId>
<version>5.0.1</version> <version>4.5.2</version>
<packaging>jar</packaging> <packaging>jar</packaging>
<properties> <properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<skipTests>true</skipTests> <skipTests>true</skipTests>
<altcoinj.version>7dc8c6f</altcoinj.version>
<altcoinj.version>7dc8c6f</altcoinj.version> <altcoinj.version>7dc8c6f</altcoinj.version>
<bitcoinj.version>0.15.10</bitcoinj.version> <bitcoinj.version>0.15.10</bitcoinj.version>
<bouncycastle.version>1.70</bouncycastle.version> <bouncycastle.version>1.70</bouncycastle.version>
@ -17,26 +16,26 @@
<ciyam-at.version>1.4.2</ciyam-at.version> <ciyam-at.version>1.4.2</ciyam-at.version>
<commons-net.version>3.8.0</commons-net.version> <commons-net.version>3.8.0</commons-net.version>
<commons-text.version>1.12.0</commons-text.version> <commons-text.version>1.12.0</commons-text.version>
<commons-io.version>2.18.0</commons-io.version> <commons-io.version>2.16.1</commons-io.version>
<commons-compress.version>1.27.1</commons-compress.version> <commons-compress.version>1.26.2</commons-compress.version>
<commons-lang3.version>3.17.0</commons-lang3.version> <commons-lang3.version>3.14.0</commons-lang3.version>
<dagger.version>1.2.2</dagger.version> <dagger.version>1.2.2</dagger.version>
<extendedset.version>0.12.3</extendedset.version> <extendedset.version>0.12.3</extendedset.version>
<git-commit-id-plugin.version>4.9.10</git-commit-id-plugin.version> <git-commit-id-plugin.version>4.9.10</git-commit-id-plugin.version>
<grpc.version>1.68.1</grpc.version> <grpc.version>1.65.0</grpc.version>
<guava.version>33.3.1-jre</guava.version> <guava.version>33.2.1-jre</guava.version>
<hamcrest-library.version>2.2</hamcrest-library.version> <hamcrest-library.version>2.2</hamcrest-library.version>
<homoglyph.version>1.2.1</homoglyph.version> <homoglyph.version>1.2.1</homoglyph.version>
<hsqldb.version>2.7.4</hsqldb.version> <hsqldb.version>2.5.1</hsqldb.version>
<icu4j.version>76.1</icu4j.version> <icu4j.version>75.1</icu4j.version>
<java-diff-utils.version>4.15</java-diff-utils.version> <java-diff-utils.version>4.12</java-diff-utils.version>
<javax.servlet-api.version>4.0.1</javax.servlet-api.version> <javax.servlet-api.version>4.0.1</javax.servlet-api.version>
<jaxb-runtime.version>2.3.9</jaxb-runtime.version> <jaxb-runtime.version>2.3.9</jaxb-runtime.version>
<jersey.version>2.42</jersey.version> <jersey.version>2.42</jersey.version>
<jetty.version>9.4.56.v20240826</jetty.version> <jetty.version>9.4.54.v20240208</jetty.version>
<json-simple.version>1.1.1</json-simple.version> <json-simple.version>1.1.1</json-simple.version>
<json.version>20240303</json.version> <json.version>20240303</json.version>
<jsoup.version>1.18.1</jsoup.version> <jsoup.version>1.17.2</jsoup.version>
<junit-jupiter-engine.version>5.11.0-M2</junit-jupiter-engine.version> <junit-jupiter-engine.version>5.11.0-M2</junit-jupiter-engine.version>
<lifecycle-mapping.version>1.0.0</lifecycle-mapping.version> <lifecycle-mapping.version>1.0.0</lifecycle-mapping.version>
<log4j.version>2.23.1</log4j.version> <log4j.version>2.23.1</log4j.version>
@ -46,20 +45,19 @@
<maven-dependency-plugin.version>3.6.1</maven-dependency-plugin.version> <maven-dependency-plugin.version>3.6.1</maven-dependency-plugin.version>
<maven-jar-plugin.version>3.4.2</maven-jar-plugin.version> <maven-jar-plugin.version>3.4.2</maven-jar-plugin.version>
<maven-package-info-plugin.version>1.1.0</maven-package-info-plugin.version> <maven-package-info-plugin.version>1.1.0</maven-package-info-plugin.version>
<maven-plugin.version>2.18.0</maven-plugin.version> <maven-plugin.version>2.16.2</maven-plugin.version>
<maven-reproducible-build-plugin.version>0.17</maven-reproducible-build-plugin.version> <maven-reproducible-build-plugin.version>0.16</maven-reproducible-build-plugin.version>
<maven-resources-plugin.version>3.3.1</maven-resources-plugin.version> <maven-resources-plugin.version>3.3.1</maven-resources-plugin.version>
<maven-shade-plugin.version>3.6.0</maven-shade-plugin.version> <maven-shade-plugin.version>3.6.0</maven-shade-plugin.version>
<maven-install-plugin.version>3.1.3</maven-install-plugin.version> <maven-surefire-plugin.version>3.3.0</maven-surefire-plugin.version>
<maven-surefire-plugin.version>3.5.2</maven-surefire-plugin.version>
<protobuf.version>3.25.3</protobuf.version> <protobuf.version>3.25.3</protobuf.version>
<replacer.version>1.5.3</replacer.version> <replacer.version>1.5.3</replacer.version>
<simplemagic.version>1.17</simplemagic.version> <simplemagic.version>1.17</simplemagic.version>
<slf4j.version>1.7.36</slf4j.version> <slf4j.version>1.7.36</slf4j.version>
<swagger-api.version>2.0.10</swagger-api.version> <swagger-api.version>2.0.10</swagger-api.version>
<swagger-ui.version>5.18.2</swagger-ui.version> <swagger-ui.version>5.17.14</swagger-ui.version>
<upnp.version>1.2</upnp.version> <upnp.version>1.2</upnp.version>
<xz.version>1.10</xz.version> <xz.version>1.9</xz.version>
</properties> </properties>
<build> <build>
<sourceDirectory>src/main/java</sourceDirectory> <sourceDirectory>src/main/java</sourceDirectory>
@ -305,30 +303,6 @@
</archive> </archive>
</configuration> </configuration>
</plugin> </plugin>
<!-- Copy modified hsqldb.jar to install / modified MANIFEST.MF-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>${maven-install-plugin.version}</version>
<configuration>
<groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<version>${hsqldb.version}</version>
<packaging>jar</packaging>
</configuration>
<executions>
<execution>
<phase>install</phase>
<goals>
<goal>install-file</goal>
</goals>
<configuration>
<file>${project.basedir}/lib/org/hsqldb/hsqldb/${hsqldb.version}/hsqldb.jar</file>
</configuration>
</execution>
</executions>
</plugin>
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId> <artifactId>maven-shade-plugin</artifactId>
@ -476,7 +450,7 @@
<scope>provided</scope> <scope>provided</scope>
<!-- needed for build, not for runtime --> <!-- needed for build, not for runtime -->
</dependency> </dependency>
<!-- HSQLDB for repository should use local version with Sealed: false --> <!-- HSQLDB for repository -->
<dependency> <dependency>
<groupId>org.hsqldb</groupId> <groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId> <artifactId>hsqldb</artifactId>
@ -747,12 +721,12 @@
<!-- BouncyCastle for crypto, including TLS secure networking --> <!-- BouncyCastle for crypto, including TLS secure networking -->
<dependency> <dependency>
<groupId>org.bouncycastle</groupId> <groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15to18</artifactId> <artifactId>bcprov-jdk15on</artifactId>
<version>${bouncycastle.version}</version> <version>${bouncycastle.version}</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.bouncycastle</groupId> <groupId>org.bouncycastle</groupId>
<artifactId>bctls-jdk15to18</artifactId> <artifactId>bctls-jdk15on</artifactId>
<version>${bouncycastle.version}</version> <version>${bouncycastle.version}</version>
</dependency> </dependency>
<dependency> <dependency>
@ -796,10 +770,5 @@
<artifactId>jaxb-runtime</artifactId> <artifactId>jaxb-runtime</artifactId>
<version>${jaxb-runtime.version}</version> <version>${jaxb-runtime.version}</version>
</dependency> </dependency>
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-core</artifactId>
<version>3.1.0</version>
</dependency>
</dependencies> </dependencies>
</project> </project>

View File

@ -1,173 +0,0 @@
package org.hsqldb.jdbc;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.hsqldb.jdbc.pool.JDBCPooledConnection;
import org.qortal.data.system.DbConnectionInfo;
import org.qortal.repository.hsqldb.HSQLDBRepositoryFactory;
import javax.sql.ConnectionEvent;
import javax.sql.PooledConnection;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Comparator;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
/**
* Class HSQLDBPoolMonitored
*
* This class uses the same logic as HSQLDBPool. The only difference is it monitors the state of every connection
* to the database. This is used for debugging purposes only.
*/
public class HSQLDBPoolMonitored extends HSQLDBPool {
private static final Logger LOGGER = LogManager.getLogger(HSQLDBRepositoryFactory.class);
private static final String EMPTY = "Empty";
private static final String AVAILABLE = "Available";
private static final String ALLOCATED = "Allocated";
private ConcurrentHashMap<Integer, DbConnectionInfo> infoByIndex;
public HSQLDBPoolMonitored(int poolSize) {
super(poolSize);
this.infoByIndex = new ConcurrentHashMap<>(poolSize);
}
/**
* Tries to retrieve a new connection using the properties that have already been
* set.
*
* @return a connection to the data source, or null if no spare connections in pool
* @exception SQLException if a database access error occurs
*/
public Connection tryConnection() throws SQLException {
for (int i = 0; i < states.length(); i++) {
if (states.compareAndSet(i, RefState.available, RefState.allocated)) {
JDBCPooledConnection pooledConnection = connections[i];
if (pooledConnection == null)
// Probably shutdown situation
return null;
infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
return pooledConnection.getConnection();
}
if (states.compareAndSet(i, RefState.empty, RefState.allocated)) {
try {
JDBCPooledConnection pooledConnection = (JDBCPooledConnection) source.getPooledConnection();
if (pooledConnection == null)
// Probably shutdown situation
return null;
pooledConnection.addConnectionEventListener(this);
pooledConnection.addStatementEventListener(this);
connections[i] = pooledConnection;
infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
return pooledConnection.getConnection();
} catch (SQLException e) {
states.set(i, RefState.empty);
infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), EMPTY));
}
}
}
return null;
}
public Connection getConnection() throws SQLException {
int var1 = 300;
if (this.source.loginTimeout != 0) {
var1 = this.source.loginTimeout * 10;
}
if (this.closed) {
throw new SQLException("connection pool is closed");
} else {
for(int var2 = 0; var2 < var1; ++var2) {
for(int var3 = 0; var3 < this.states.length(); ++var3) {
if (this.states.compareAndSet(var3, 1, 2)) {
infoByIndex.put(var3, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
return this.connections[var3].getConnection();
}
if (this.states.compareAndSet(var3, 0, 2)) {
try {
JDBCPooledConnection var4 = (JDBCPooledConnection)this.source.getPooledConnection();
var4.addConnectionEventListener(this);
var4.addStatementEventListener(this);
this.connections[var3] = var4;
infoByIndex.put(var3, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
return this.connections[var3].getConnection();
} catch (SQLException var6) {
this.states.set(var3, 0);
infoByIndex.put(var3, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), EMPTY));
}
}
}
try {
Thread.sleep(100L);
} catch (InterruptedException var5) {
}
}
throw JDBCUtil.invalidArgument();
}
}
public void connectionClosed(ConnectionEvent event) {
PooledConnection connection = (PooledConnection) event.getSource();
for (int i = 0; i < connections.length; i++) {
if (connections[i] == connection) {
states.set(i, RefState.available);
infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), AVAILABLE));
break;
}
}
}
public void connectionErrorOccurred(ConnectionEvent event) {
PooledConnection connection = (PooledConnection) event.getSource();
for (int i = 0; i < connections.length; i++) {
if (connections[i] == connection) {
states.set(i, RefState.allocated);
connections[i] = null;
states.set(i, RefState.empty);
infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), EMPTY));
break;
}
}
}
public List<DbConnectionInfo> getDbConnectionsStates() {
return infoByIndex.values().stream()
.sorted(Comparator.comparingLong(DbConnectionInfo::getUpdated))
.collect(Collectors.toList());
}
private int findConnectionIndex(ConnectionEvent connectionEvent) {
PooledConnection pooledConnection = (PooledConnection) connectionEvent.getSource();
for(int i = 0; i < this.connections.length; ++i) {
if (this.connections[i] == pooledConnection) {
return i;
}
}
return -1;
}
}

View File

@ -1,17 +1,14 @@
package org.qortal; package org.qortal;
import org.apache.commons.io.FileUtils;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.bouncycastle.jce.provider.BouncyCastleProvider; import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.jsse.provider.BouncyCastleJsseProvider; import org.bouncycastle.jsse.provider.BouncyCastleJsseProvider;
import org.qortal.api.ApiKey; import org.qortal.api.ApiKey;
import org.qortal.api.ApiRequest; import org.qortal.api.ApiRequest;
import org.qortal.controller.Controller;
import org.qortal.controller.RestartNode; import org.qortal.controller.RestartNode;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import java.io.File;
import java.io.IOException; import java.io.IOException;
import java.lang.management.ManagementFactory; import java.lang.management.ManagementFactory;
import java.nio.file.Files; import java.nio.file.Files;
@ -19,8 +16,6 @@ import java.nio.file.Path;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.security.Security; import java.security.Security;
import java.util.*; import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import static org.qortal.controller.RestartNode.AGENTLIB_JVM_HOLDER_ARG; import static org.qortal.controller.RestartNode.AGENTLIB_JVM_HOLDER_ARG;
@ -43,7 +38,7 @@ public class ApplyRestart {
private static final String JAVA_TOOL_OPTIONS_NAME = "JAVA_TOOL_OPTIONS"; private static final String JAVA_TOOL_OPTIONS_NAME = "JAVA_TOOL_OPTIONS";
private static final String JAVA_TOOL_OPTIONS_VALUE = ""; private static final String JAVA_TOOL_OPTIONS_VALUE = "";
private static final long CHECK_INTERVAL = 30 * 1000L; // ms private static final long CHECK_INTERVAL = 10 * 1000L; // ms
private static final int MAX_ATTEMPTS = 12; private static final int MAX_ATTEMPTS = 12;
public static void main(String[] args) { public static void main(String[] args) {
@ -56,38 +51,21 @@ public class ApplyRestart {
else else
Settings.getInstance(); Settings.getInstance();
LOGGER.info("Applying restart this can take up to 5 minutes..."); LOGGER.info("Applying restart...");
// Shutdown node using API // Shutdown node using API
if (!shutdownNode()) if (!shutdownNode())
return; return;
try { // Restart node
// Give some time for shutdown restartNode(args);
TimeUnit.SECONDS.sleep(60);
// Remove blockchain lock if exist LOGGER.info("Restarting...");
ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
if (blockchainLock.isLocked())
blockchainLock.unlock();
// Remove blockchain lock file if still exist
TimeUnit.SECONDS.sleep(60);
deleteLock();
// Restart node
TimeUnit.SECONDS.sleep(15);
restartNode(args);
LOGGER.info("Restarting...");
} catch (InterruptedException e) {
LOGGER.error("Unable to restart", e);
}
} }
private static boolean shutdownNode() { private static boolean shutdownNode() {
String baseUri = "http://localhost:" + Settings.getInstance().getApiPort() + "/"; String baseUri = "http://localhost:" + Settings.getInstance().getApiPort() + "/";
LOGGER.debug(() -> String.format("Shutting down node using API via %s", baseUri)); LOGGER.info(() -> String.format("Shutting down node using API via %s", baseUri));
// The /admin/stop endpoint requires an API key, which may or may not be already generated // The /admin/stop endpoint requires an API key, which may or may not be already generated
boolean apiKeyNewlyGenerated = false; boolean apiKeyNewlyGenerated = false;
@ -117,17 +95,10 @@ public class ApplyRestart {
String response = ApiRequest.perform(baseUri + "admin/stop", params); String response = ApiRequest.perform(baseUri + "admin/stop", params);
if (response == null) { if (response == null) {
// No response - consider node shut down // No response - consider node shut down
try {
TimeUnit.SECONDS.sleep(30);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
if (apiKeyNewlyGenerated) { if (apiKeyNewlyGenerated) {
// API key was newly generated for restarting node, so we need to remove it // API key was newly generated for restarting node, so we need to remove it
ApplyRestart.removeGeneratedApiKey(); ApplyRestart.removeGeneratedApiKey();
} }
return true; return true;
} }
@ -163,22 +134,7 @@ public class ApplyRestart {
apiKey.delete(); apiKey.delete();
} catch (IOException e) { } catch (IOException e) {
LOGGER.error("Error loading or deleting API key: {}", e.getMessage()); LOGGER.info("Error loading or deleting API key: {}", e.getMessage());
}
}
private static void deleteLock() {
// Get the repository path from settings
String repositoryPath = Settings.getInstance().getRepositoryPath();
LOGGER.debug(String.format("Repository path is: %s", repositoryPath));
try {
Path root = Paths.get(repositoryPath);
File lockFile = new File(root.resolve("blockchain.lck").toUri());
LOGGER.debug("Lockfile is: {}", lockFile);
FileUtils.forceDelete(FileUtils.getFile(lockFile));
} catch (IOException e) {
LOGGER.debug("Error deleting blockchain lock file: {}", e.getMessage());
} }
} }
@ -194,10 +150,9 @@ public class ApplyRestart {
List<String> javaCmd; List<String> javaCmd;
if (Files.exists(exeLauncher)) { if (Files.exists(exeLauncher)) {
javaCmd = List.of(exeLauncher.toString()); javaCmd = Arrays.asList(exeLauncher.toString());
} else { } else {
javaCmd = new ArrayList<>(); javaCmd = new ArrayList<>();
// Java runtime binary itself // Java runtime binary itself
javaCmd.add(javaBinary.toString()); javaCmd.add(javaBinary.toString());

View File

@ -2,31 +2,19 @@ package org.qortal.account;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.qortal.api.resource.TransactionsResource;
import org.qortal.block.BlockChain; import org.qortal.block.BlockChain;
import org.qortal.controller.LiteNode; import org.qortal.controller.LiteNode;
import org.qortal.data.account.AccountBalanceData; import org.qortal.data.account.AccountBalanceData;
import org.qortal.data.account.AccountData; import org.qortal.data.account.AccountData;
import org.qortal.data.account.RewardShareData; import org.qortal.data.account.RewardShareData;
import org.qortal.data.naming.NameData;
import org.qortal.data.transaction.TransactionData;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
import org.qortal.repository.GroupRepository;
import org.qortal.repository.NameRepository;
import org.qortal.repository.Repository; import org.qortal.repository.Repository;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import org.qortal.utils.Groups;
import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType; import javax.xml.bind.annotation.XmlAccessorType;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import static org.qortal.utils.Amounts.prettyAmount; import static org.qortal.utils.Amounts.prettyAmount;
@XmlAccessorType(XmlAccessType.NONE) // Stops JAX-RS errors when unmarshalling blockchain config @XmlAccessorType(XmlAccessType.NONE) // Stops JAX-RS errors when unmarshalling blockchain config
@ -205,85 +193,27 @@ public class Account {
/** Returns whether account can be considered a "minting account". /** Returns whether account can be considered a "minting account".
* <p> * <p>
* To be considered a "minting account", the account needs to pass some of these tests:<br> * To be considered a "minting account", the account needs to pass at least one of these tests:<br>
* <ul> * <ul>
* <li>account's level is at least <tt>minAccountLevelToMint</tt> from blockchain config</li> * <li>account's level is at least <tt>minAccountLevelToMint</tt> from blockchain config</li>
* <li>account's address has registered a name</li> * <li>account has 'founder' flag set</li>
* <li>account's address is a member of the minter group</li>
* </ul> * </ul>
* *
* @param isGroupValidated true if this account has already been validated for MINTER Group membership
* @return true if account can be considered "minting account" * @return true if account can be considered "minting account"
* @throws DataException * @throws DataException
*/ */
public boolean canMint(boolean isGroupValidated) throws DataException { public boolean canMint() throws DataException {
AccountData accountData = this.repository.getAccountRepository().getAccount(this.address); AccountData accountData = this.repository.getAccountRepository().getAccount(this.address);
NameRepository nameRepository = this.repository.getNameRepository(); if (accountData == null)
GroupRepository groupRepository = this.repository.getGroupRepository(); return false;
String myAddress = accountData.getAddress();
int blockchainHeight = this.repository.getBlockRepository().getBlockchainHeight(); Integer level = accountData.getLevel();
if (level != null && level >= BlockChain.getInstance().getMinAccountLevelToMint())
return true;
int levelToMint; // Founders can always mint, unless they have a penalty
if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0)
if( blockchainHeight >= BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() ) { return true;
levelToMint = 0;
}
else {
levelToMint = BlockChain.getInstance().getMinAccountLevelToMint();
}
int level = accountData.getLevel();
List<Integer> groupIdsToMint = Groups.getGroupIdsToMint( BlockChain.getInstance(), blockchainHeight );
int nameCheckHeight = BlockChain.getInstance().getOnlyMintWithNameHeight();
int groupCheckHeight = BlockChain.getInstance().getGroupMemberCheckHeight();
int removeNameCheckHeight = BlockChain.getInstance().getRemoveOnlyMintWithNameHeight();
// Can only mint if:
// Account's level is at least minAccountLevelToMint from blockchain config
if (blockchainHeight < nameCheckHeight) {
if (Account.isFounder(accountData.getFlags())) {
return accountData.getBlocksMintedPenalty() == 0;
} else {
return level >= levelToMint;
}
}
// Can only mint on onlyMintWithNameHeight from blockchain config if:
// Account's level is at least minAccountLevelToMint from blockchain config
// Account's address has registered a name
if (blockchainHeight >= nameCheckHeight && blockchainHeight < groupCheckHeight) {
List<NameData> myName = nameRepository.getNamesByOwner(myAddress);
if (Account.isFounder(accountData.getFlags())) {
return accountData.getBlocksMintedPenalty() == 0 && !myName.isEmpty();
} else {
return level >= levelToMint && !myName.isEmpty();
}
}
// Can only mint on groupMemberCheckHeight from blockchain config if:
// Account's level is at least minAccountLevelToMint from blockchain config
// Account's address has registered a name
// Account's address is a member of the minter group
if (blockchainHeight >= groupCheckHeight && blockchainHeight < removeNameCheckHeight) {
List<NameData> myName = nameRepository.getNamesByOwner(myAddress);
if (Account.isFounder(accountData.getFlags())) {
return accountData.getBlocksMintedPenalty() == 0 && !myName.isEmpty() && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
} else {
return level >= levelToMint && !myName.isEmpty() && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
}
}
// Can only mint on removeOnlyMintWithNameHeight from blockchain config if:
// Account's level is at least minAccountLevelToMint from blockchain config
// Account's address is a member of the minter group
if (blockchainHeight >= removeNameCheckHeight) {
if (Account.isFounder(accountData.getFlags())) {
return accountData.getBlocksMintedPenalty() == 0 && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
} else {
return level >= levelToMint && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
}
}
return false; return false;
} }
@ -298,6 +228,7 @@ public class Account {
return this.repository.getAccountRepository().getBlocksMintedPenaltyCount(this.address); return this.repository.getAccountRepository().getBlocksMintedPenaltyCount(this.address);
} }
/** Returns whether account can build reward-shares. /** Returns whether account can build reward-shares.
* <p> * <p>
* To be able to create reward-shares, the account needs to pass at least one of these tests:<br> * To be able to create reward-shares, the account needs to pass at least one of these tests:<br>
@ -311,7 +242,6 @@ public class Account {
*/ */
public boolean canRewardShare() throws DataException { public boolean canRewardShare() throws DataException {
AccountData accountData = this.repository.getAccountRepository().getAccount(this.address); AccountData accountData = this.repository.getAccountRepository().getAccount(this.address);
if (accountData == null) if (accountData == null)
return false; return false;
@ -322,9 +252,6 @@ public class Account {
if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0) if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0)
return true; return true;
if( this.repository.getBlockRepository().getBlockchainHeight() >= BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() )
return true;
return false; return false;
} }
@ -367,160 +294,6 @@ public class Account {
return accountData.getLevel(); return accountData.getLevel();
} }
/**
* Get Primary Name
*
* @return the primary name for this address if present, otherwise empty
*
* @throws DataException
*/
public Optional<String> getPrimaryName() throws DataException {
return this.repository.getNameRepository().getPrimaryName(this.address);
}
/**
* Remove Primary Name
*
* @throws DataException
*/
public void removePrimaryName() throws DataException {
this.repository.getNameRepository().removePrimaryName(this.address);
}
/**
* Reset Primary Name
*
* Set primary name based on the names (and their history) this account owns.
*
* @param confirmationStatus the status of the transactions for the determining the primary name
*
* @return the primary name, empty if their isn't one
*
* @throws DataException
*/
public Optional<String> resetPrimaryName(TransactionsResource.ConfirmationStatus confirmationStatus) throws DataException {
Optional<String> primaryName = determinePrimaryName(confirmationStatus);
if(primaryName.isPresent()) {
return setPrimaryName(primaryName.get());
}
else {
return primaryName;
}
}
/**
* Determine Primary Name
*
* Determine primary name based on a list of registered names.
*
* @param confirmationStatus the status of the transactions for this determination
*
* @return the primary name, empty if there is no primary name
*
* @throws DataException
*/
public Optional<String> determinePrimaryName(TransactionsResource.ConfirmationStatus confirmationStatus) throws DataException {
// all registered names for the owner
List<NameData> names = this.repository.getNameRepository().getNamesByOwner(this.address);
Optional<String> primaryName;
// if no registered names, the no primary name possible
if (names.isEmpty()) {
primaryName = Optional.empty();
}
// if names
else {
// if one name, then that is the primary name
if (names.size() == 1) {
primaryName = Optional.of( names.get(0).getName() );
}
// if more than one name, then seek the earliest name acquisition that was never released
else {
Map<String, TransactionData> txByName = new HashMap<>(names.size());
// for each name, get the latest transaction
for (NameData nameData : names) {
// since the name is currently registered to the owner,
// we assume the latest transaction involving this name was the transaction that the acquired
// name through registration, purchase or update
Optional<TransactionData> latestTransaction
= this.repository
.getTransactionRepository()
.getTransactionsInvolvingName(
nameData.getName(),
confirmationStatus
)
.stream()
.sorted(Comparator.comparing(
TransactionData::getTimestamp).reversed()
)
.findFirst(); // first is the last, since it was reversed
// if there is a latest transaction, expected for all registered names
if (latestTransaction.isPresent()) {
txByName.put(nameData.getName(), latestTransaction.get());
}
// if there is no latest transaction, then
else {
LOGGER.warn("No matching transaction for name: " + nameData.getName());
}
}
// get the first name aqcuistion for this address
Optional<Map.Entry<String, TransactionData>> firstNameEntry
= txByName.entrySet().stream().sorted(Comparator.comparing(entry -> entry.getValue().getTimestamp())).findFirst();
// if their is a name acquisition, then the first one is the primary name
if (firstNameEntry.isPresent()) {
primaryName = Optional.of( firstNameEntry.get().getKey() );
}
// if there is no nameacquistion, then there is no primary name
else {
primaryName = Optional.empty();
}
}
}
return primaryName;
}
/**
* Set Primary Name
*
* @param primaryName the primary to set to this address
*
* @return the primary name if successful, empty if unsuccessful
*
* @throws DataException
*/
public Optional<String> setPrimaryName( String primaryName ) throws DataException {
int changed = this.repository.getNameRepository().setPrimaryName(this.address, primaryName);
return changed > 0 ? Optional.of(primaryName) : Optional.empty();
}
/**
* Returns reward-share minting address, or unknown if reward-share does not exist.
*
* @param repository
* @param rewardSharePublicKey
* @return address or unknown
* @throws DataException
*/
public static String getRewardShareMintingAddress(Repository repository, byte[] rewardSharePublicKey) throws DataException {
// Find actual minter address
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(rewardSharePublicKey);
if (rewardShareData == null)
return "Unknown";
return rewardShareData.getMinter();
}
/** /**
* Returns 'effective' minting level, or zero if reward-share does not exist. * Returns 'effective' minting level, or zero if reward-share does not exist.
* *
@ -538,7 +311,6 @@ public class Account {
Account rewardShareMinter = new Account(repository, rewardShareData.getMinter()); Account rewardShareMinter = new Account(repository, rewardShareData.getMinter());
return rewardShareMinter.getEffectiveMintingLevel(); return rewardShareMinter.getEffectiveMintingLevel();
} }
/** /**
* Returns 'effective' minting level, with a fix for the zero level. * Returns 'effective' minting level, with a fix for the zero level.
* <p> * <p>

View File

@ -46,7 +46,6 @@ public class ApiService {
private ApiService() { private ApiService() {
this.config = new ResourceConfig(); this.config = new ResourceConfig();
this.config.packages("org.qortal.api.resource", "org.qortal.api.restricted.resource"); this.config.packages("org.qortal.api.resource", "org.qortal.api.restricted.resource");
this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
this.config.register(OpenApiResource.class); this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class); this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class); this.config.register(AnnotationPostProcessor.class);
@ -195,10 +194,8 @@ public class ApiService {
context.addServlet(AdminStatusWebSocket.class, "/websockets/admin/status"); context.addServlet(AdminStatusWebSocket.class, "/websockets/admin/status");
context.addServlet(BlocksWebSocket.class, "/websockets/blocks"); context.addServlet(BlocksWebSocket.class, "/websockets/blocks");
context.addServlet(DataMonitorSocket.class, "/websockets/datamonitor");
context.addServlet(ActiveChatsWebSocket.class, "/websockets/chat/active/*"); context.addServlet(ActiveChatsWebSocket.class, "/websockets/chat/active/*");
context.addServlet(ChatMessagesWebSocket.class, "/websockets/chat/messages"); context.addServlet(ChatMessagesWebSocket.class, "/websockets/chat/messages");
context.addServlet(UnsignedFeesSocket.class, "/websockets/crosschain/unsignedfees");
context.addServlet(TradeOffersWebSocket.class, "/websockets/crosschain/tradeoffers"); context.addServlet(TradeOffersWebSocket.class, "/websockets/crosschain/tradeoffers");
context.addServlet(TradeBotWebSocket.class, "/websockets/crosschain/tradebot"); context.addServlet(TradeBotWebSocket.class, "/websockets/crosschain/tradebot");
context.addServlet(TradePresenceWebSocket.class, "/websockets/crosschain/tradepresence"); context.addServlet(TradePresenceWebSocket.class, "/websockets/crosschain/tradepresence");

View File

@ -40,7 +40,6 @@ public class DevProxyService {
private DevProxyService() { private DevProxyService() {
this.config = new ResourceConfig(); this.config = new ResourceConfig();
this.config.packages("org.qortal.api.proxy.resource", "org.qortal.api.resource"); this.config.packages("org.qortal.api.proxy.resource", "org.qortal.api.resource");
this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
this.config.register(OpenApiResource.class); this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class); this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class); this.config.register(AnnotationPostProcessor.class);

View File

@ -39,7 +39,6 @@ public class DomainMapService {
private DomainMapService() { private DomainMapService() {
this.config = new ResourceConfig(); this.config = new ResourceConfig();
this.config.packages("org.qortal.api.resource", "org.qortal.api.domainmap.resource"); this.config.packages("org.qortal.api.resource", "org.qortal.api.domainmap.resource");
this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
this.config.register(OpenApiResource.class); this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class); this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class); this.config.register(AnnotationPostProcessor.class);

View File

@ -39,7 +39,6 @@ public class GatewayService {
private GatewayService() { private GatewayService() {
this.config = new ResourceConfig(); this.config = new ResourceConfig();
this.config.packages("org.qortal.api.resource", "org.qortal.api.gateway.resource"); this.config.packages("org.qortal.api.resource", "org.qortal.api.gateway.resource");
this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
this.config.register(OpenApiResource.class); this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class); this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class); this.config.register(AnnotationPostProcessor.class);

View File

@ -1,13 +1,14 @@
package org.qortal.api; package org.qortal.api;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.jsoup.Jsoup; import org.jsoup.Jsoup;
import org.jsoup.nodes.Document; import org.jsoup.nodes.Document;
import org.jsoup.select.Elements; import org.jsoup.select.Elements;
import org.qortal.arbitrary.misc.Service; import org.qortal.arbitrary.misc.Service;
import java.util.Objects; import java.util.Objects;
public class HTMLParser { public class HTMLParser {
private static final Logger LOGGER = LogManager.getLogger(HTMLParser.class); private static final Logger LOGGER = LogManager.getLogger(HTMLParser.class);
@ -21,11 +22,10 @@ public class HTMLParser {
private String identifier; private String identifier;
private String path; private String path;
private String theme; private String theme;
private String lang;
private boolean usingCustomRouting; private boolean usingCustomRouting;
public HTMLParser(String resourceId, String inPath, String prefix, boolean includeResourceIdInPrefix, byte[] data, public HTMLParser(String resourceId, String inPath, String prefix, boolean includeResourceIdInPrefix, byte[] data,
String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting, String lang) { String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting) {
String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : String.format("/%s",inPath); String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : String.format("/%s",inPath);
this.qdnBase = includeResourceIdInPrefix ? String.format("%s/%s", prefix, resourceId) : prefix; this.qdnBase = includeResourceIdInPrefix ? String.format("%s/%s", prefix, resourceId) : prefix;
this.qdnBaseWithPath = includeResourceIdInPrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : String.format("%s%s", prefix, inPathWithoutFilename); this.qdnBaseWithPath = includeResourceIdInPrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : String.format("%s%s", prefix, inPathWithoutFilename);
@ -36,7 +36,6 @@ public class HTMLParser {
this.identifier = identifier; this.identifier = identifier;
this.path = inPath; this.path = inPath;
this.theme = theme; this.theme = theme;
this.lang = lang;
this.usingCustomRouting = usingCustomRouting; this.usingCustomRouting = usingCustomRouting;
} }
@ -62,13 +61,9 @@ public class HTMLParser {
String identifier = this.identifier != null ? this.identifier.replace("\\", "").replace("\"","\\\"") : ""; String identifier = this.identifier != null ? this.identifier.replace("\\", "").replace("\"","\\\"") : "";
String path = this.path != null ? this.path.replace("\\", "").replace("\"","\\\"") : ""; String path = this.path != null ? this.path.replace("\\", "").replace("\"","\\\"") : "";
String theme = this.theme != null ? this.theme.replace("\\", "").replace("\"","\\\"") : ""; String theme = this.theme != null ? this.theme.replace("\\", "").replace("\"","\\\"") : "";
String lang = this.lang != null ? this.lang.replace("\\", "").replace("\"", "\\\"") : "";
String qdnBase = this.qdnBase != null ? this.qdnBase.replace("\\", "").replace("\"","\\\"") : ""; String qdnBase = this.qdnBase != null ? this.qdnBase.replace("\\", "").replace("\"","\\\"") : "";
String qdnBaseWithPath = this.qdnBaseWithPath != null ? this.qdnBaseWithPath.replace("\\", "").replace("\"","\\\"") : ""; String qdnBaseWithPath = this.qdnBaseWithPath != null ? this.qdnBaseWithPath.replace("\\", "").replace("\"","\\\"") : "";
String qdnContextVar = String.format( String qdnContextVar = String.format("<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\"; var _qdnBase=\"%s\"; var _qdnBaseWithPath=\"%s\";</script>", qdnContext, theme, service, name, identifier, path, qdnBase, qdnBaseWithPath);
"<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnLang=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\"; var _qdnBase=\"%s\"; var _qdnBaseWithPath=\"%s\";</script>",
qdnContext, theme, lang, service, name, identifier, path, qdnBase, qdnBaseWithPath
);
head.get(0).prepend(qdnContextVar); head.get(0).prepend(qdnContextVar);
// Add base href tag // Add base href tag

View File

@ -1,13 +1,7 @@
package org.qortal.api.model; package org.qortal.api.model;
import org.qortal.account.Account;
import org.qortal.repository.DataException;
import org.qortal.repository.RepositoryManager;
import org.qortal.repository.Repository;
import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType; import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
// All properties to be converted to JSON via JAXB // All properties to be converted to JSON via JAXB
@XmlAccessorType(XmlAccessType.FIELD) @XmlAccessorType(XmlAccessType.FIELD)
@ -53,31 +47,4 @@ public class ApiOnlineAccount {
return this.recipientAddress; return this.recipientAddress;
} }
public int getMinterLevelFromPublicKey() {
try (final Repository repository = RepositoryManager.getRepository()) {
return Account.getRewardShareEffectiveMintingLevel(repository, this.rewardSharePublicKey);
} catch (DataException e) {
return 0;
}
}
public boolean getIsMember() {
try (final Repository repository = RepositoryManager.getRepository()) {
return repository.getGroupRepository().memberExists(694, getMinterAddress());
} catch (DataException e) {
return false;
}
}
// JAXB special
@XmlElement(name = "minterLevel")
protected int getMinterLevel() {
return getMinterLevelFromPublicKey();
}
@XmlElement(name = "isMinterMember")
protected boolean getMinterMember() {
return getIsMember();
}
} }

View File

@ -9,7 +9,6 @@ import java.math.BigInteger;
public class BlockMintingInfo { public class BlockMintingInfo {
public byte[] minterPublicKey; public byte[] minterPublicKey;
public String minterAddress;
public int minterLevel; public int minterLevel;
public int onlineAccountsCount; public int onlineAccountsCount;
public BigDecimal maxDistance; public BigDecimal maxDistance;
@ -20,4 +19,5 @@ public class BlockMintingInfo {
public BlockMintingInfo() { public BlockMintingInfo() {
} }
} }

View File

@ -1,72 +0,0 @@
package org.qortal.api.model;
import io.swagger.v3.oas.annotations.media.Schema;
import org.qortal.data.crosschain.CrossChainTradeData;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
// All properties to be converted to JSON via JAXB
@XmlAccessorType(XmlAccessType.FIELD)
public class CrossChainTradeLedgerEntry {
private String market;
private String currency;
@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
private long quantity;
@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
private long feeAmount;
private String feeCurrency;
@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
private long totalPrice;
private long tradeTimestamp;
protected CrossChainTradeLedgerEntry() {
/* For JAXB */
}
public CrossChainTradeLedgerEntry(String market, String currency, long quantity, long feeAmount, String feeCurrency, long totalPrice, long tradeTimestamp) {
this.market = market;
this.currency = currency;
this.quantity = quantity;
this.feeAmount = feeAmount;
this.feeCurrency = feeCurrency;
this.totalPrice = totalPrice;
this.tradeTimestamp = tradeTimestamp;
}
public String getMarket() {
return market;
}
public String getCurrency() {
return currency;
}
public long getQuantity() {
return quantity;
}
public long getFeeAmount() {
return feeAmount;
}
public String getFeeCurrency() {
return feeCurrency;
}
public long getTotalPrice() {
return totalPrice;
}
public long getTradeTimestamp() {
return tradeTimestamp;
}
}

View File

@ -1,50 +0,0 @@
package org.qortal.api.model;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import java.util.Objects;
// All properties to be converted to JSON via JAXB
@XmlAccessorType(XmlAccessType.FIELD)
public class DatasetStatus {
private String name;
private long count;
public DatasetStatus() {}
public DatasetStatus(String name, long count) {
this.name = name;
this.count = count;
}
public String getName() {
return name;
}
public long getCount() {
return count;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
DatasetStatus that = (DatasetStatus) o;
return count == that.count && Objects.equals(name, that.name);
}
@Override
public int hashCode() {
return Objects.hash(name, count);
}
@Override
public String toString() {
return "DatasetStatus{" +
"name='" + name + '\'' +
", count=" + count +
'}';
}
}

View File

@ -1,692 +0,0 @@
package org.qortal.api.model.crosschain;
import org.qortal.crosschain.ServerInfo;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import java.util.Arrays;
@XmlAccessorType(XmlAccessType.FIELD)
public class BitcoinyTBDRequest {
/**
* Target Timespan
*
* extracted from /src/chainparams.cpp class
* consensus.nPowTargetTimespan
*/
private int targetTimespan;
/**
* Target Spacing
*
* extracted from /src/chainparams.cpp class
* consensus.nPowTargetSpacing
*/
private int targetSpacing;
/**
* Packet Magic
*
* extracted from /src/chainparams.cpp class
* Concatenate the 4 values in pchMessageStart, then convert the hex to decimal.
*
* Ex. litecoin
* pchMessageStart[0] = 0xfb;
* pchMessageStart[1] = 0xc0;
* pchMessageStart[2] = 0xb6;
* pchMessageStart[3] = 0xdb;
* packetMagic = 0xfbc0b6db = 4223710939
*/
private long packetMagic;
/**
* Port
*
* extracted from /src/chainparams.cpp class
* nDefaultPort
*/
private int port;
/**
* Address Header
*
* extracted from /src/chainparams.cpp class
* base58Prefixes[PUBKEY_ADDRESS] from Main Network
*/
private int addressHeader;
/**
* P2sh Header
*
* extracted from /src/chainparams.cpp class
* base58Prefixes[SCRIPT_ADDRESS] from Main Network
*/
private int p2shHeader;
/**
* Segwit Address Hrp
*
* HRP -> Human Readable Parts
*
* extracted from /src/chainparams.cpp class
* bech32_hrp
*/
private String segwitAddressHrp;
/**
* Dumped Private Key Header
*
* extracted from /src/chainparams.cpp class
* base58Prefixes[SECRET_KEY] from Main Network
* This is usually, but not always ... addressHeader + 128
*/
private int dumpedPrivateKeyHeader;
/**
* Subsidy Decreased Block Count
*
* extracted from /src/chainparams.cpp class
* consensus.nSubsidyHalvingInterval
*
* Digibyte does not support this, because they do halving differently.
*/
private int subsidyDecreaseBlockCount;
/**
* Expected Genesis Hash
*
* extracted from /src/chainparams.cpp class
* consensus.hashGenesisBlock
* Remove '0x' prefix
*/
private String expectedGenesisHash;
/**
* Common Script Pub Key
*
* extracted from /src/chainparams.cpp class
* This is the key commonly used to sign alerts for altcoins. Bitcoin and Digibyte are know exceptions.
*/
public static final String SCRIPT_PUB_KEY = "040184710fa689ad5023690c80f3a49c8f13f8d45b8c857fbcbc8bc4a8e4d3eb4b10f4d4604fa08dce601aaf0f470216fe1b51850b4acf21b179c45070ac7b03a9";
/**
* The Script Pub Key
*
* extracted from /src/chainparams.cpp class
* The key to sign alerts.
*
* const CScript genesisOutputScript = CScript() << ParseHex("040184710fa689ad5023690c80f3a49c8f13f8d45b8c857fbcbc8bc4a8e4d3eb4b10f4d4604fa08dce601aaf0f470216fe1b51850b4acf21b179c45070ac7b03a9") << OP_CHECKSIG;
*
* ie LTC = 040184710fa689ad5023690c80f3a49c8f13f8d45b8c857fbcbc8bc4a8e4d3eb4b10f4d4604fa08dce601aaf0f470216fe1b51850b4acf21b179c45070ac7b03a9
*
* this may be the same value as scripHex
*/
private String pubKey;
/**
* DNS Seeds
*
* extracted from /src/chainparams.cpp class
* vSeeds
*/
private String[] dnsSeeds;
/**
* BIP32 Header P2PKH Pub
*
* extracted from /src/chainparams.cpp class
* Concatenate the 4 values in base58Prefixes[EXT_PUBLIC_KEY]
* base58Prefixes[EXT_PUBLIC_KEY] = {0x04, 0x88, 0xB2, 0x1E} = 0x0488B21E
*/
private int bip32HeaderP2PKHpub;
/**
* BIP32 Header P2PKH Priv
*
* extracted from /src/chainparams.cpp class
* Concatenate the 4 values in base58Prefixes[EXT_SECRET_KEY]
* base58Prefixes[EXT_SECRET_KEY] = {0x04, 0x88, 0xAD, 0xE4} = 0x0488ADE4
*/
private int bip32HeaderP2PKHpriv;
/**
* Address Header (Testnet)
*
* extracted from /src/chainparams.cpp class
* base58Prefixes[PUBKEY_ADDRESS] from Testnet
*/
private int addressHeaderTestnet;
/**
* BIP32 Header P2PKH Pub (Testnet)
*
* extracted from /src/chainparams.cpp class
* Concatenate the 4 values in base58Prefixes[EXT_PUBLIC_KEY]
* base58Prefixes[EXT_PUBLIC_KEY] = {0x04, 0x88, 0xB2, 0x1E} = 0x0488B21E
*/
private int bip32HeaderP2PKHpubTestnet;
/**
* BIP32 Header P2PKH Priv (Testnet)
*
* extracted from /src/chainparams.cpp class
* Concatenate the 4 values in base58Prefixes[EXT_SECRET_KEY]
* base58Prefixes[EXT_SECRET_KEY] = {0x04, 0x88, 0xAD, 0xE4} = 0x0488ADE4
*/
private int bip32HeaderP2PKHprivTestnet;
/**
* Id
*
* "org.litecoin.production" for LTC
* I'm guessing this just has to match others for trading purposes.
*/
private String id;
/**
* Majority Enforce Block Upgrade
*
* All coins are setting this to 750, except DOGE is setting this to 1500.
*/
private int majorityEnforceBlockUpgrade;
/**
* Majority Reject Block Outdated
*
* All coins are setting this to 950, except DOGE is setting this to 1900.
*/
private int majorityRejectBlockOutdated;
/**
* Majority Window
*
* All coins are setting this to 1000, except DOGE is setting this to 2000.
*/
private int majorityWindow;
/**
* Code
*
* "LITE" for LTC
* Currency code for full unit.
*/
private String code;
/**
* mCode
*
* "mLITE" for LTC
* Currency code for milli unit.
*/
private String mCode;
/**
* Base Code
*
* "Liteoshi" for LTC
* Currency code for base unit.
*/
private String baseCode;
/**
* Min Non Dust Output
*
* 100000 for LTC, web search for minimum transaction fee per kB
*/
private int minNonDustOutput;
/**
* URI Scheme
*
* uriScheme = "litecoin" for LTC
* Do a web search to find this value.
*/
private String uriScheme;
/**
* Protocol Version Minimum
*
* 70002 for LTC
* extracted from /src/protocol.h class
*/
private int protocolVersionMinimum;
/**
* Protocol Version Current
*
* 70003 for LTC
* extracted from /src/protocol.h class
*/
private int protocolVersionCurrent;
/**
* Has Max Money
*
* false for DOGE, true for BTC and LTC
*/
private boolean hasMaxMoney;
/**
* Max Money
*
* 84000000 for LTC, 21000000 for BTC
* extracted from src/amount.h class
*/
private long maxMoney;
/**
* Currency Code
*
* The trading symbol, ie LTC, BTC, DOGE
*/
private String currencyCode;
/**
* Minimum Order Amount
*
* web search, LTC minimumOrderAmount = 1000000, 0.01 LTC minimum order to avoid dust errors
*/
private long minimumOrderAmount;
/**
* Fee Per Kb
*
* web search, LTC feePerKb = 10000, 0.0001 LTC per 1000 bytes
*/
private long feePerKb;
/**
* Network Name
*
* ie Litecoin-MAIN
*/
private String networkName;
/**
* Fee Required
*
* web search, LTC fee required = 1000L
*/
private long feeRequired;
/**
* Extended Public Key
*
* xpub for operations that require wallet watching
*/
private String extendedPublicKey;
/**
* Send Amount
*
* The amount to send in base units. Also, requires sending fee per byte, receiving address and sender's extended private key.
*/
private long sendAmount;
/**
* Sending Fee Per Byte
*
* The fee to include on a send request in base units. Also, requires receiving address, sender's extended private key and send amount.
*/
private long sendingFeePerByte;
/**
* Receiving Address
*
* The receiving address for a send request. Also, requires send amount, sender's extended private key and sending fee per byte.
*/
private String receivingAddress;
/**
* Extended Private Key
*
* xpriv address for a send request. Also, requires receiving address, send amount and sending fee per byte.
*/
private String extendedPrivateKey;
/**
* Server Info
*
* For adding, removing, setting current server requests.
*/
private ServerInfo serverInfo;
/**
* Script Sig
*
* extracted from /src/chainparams.cpp class
* pszTimestamp
*
* transform this value - https://bitcoin.stackexchange.com/questions/13122/scriptsig-coinbase-structure-of-the-genesis-block
* ie LTC = 04ffff001d0104404e592054696d65732030352f4f63742f32303131205374657665204a6f62732c204170706c65e280997320566973696f6e6172792c2044696573206174203536
* ie DOGE = 04ffff001d0104084e696e746f6e646f
*/
private String scriptSig;
/**
* Script Hex
*
* extracted from /src/chainparams.cpp class
* genesisOutputScript
*
* ie LTC = 040184710fa689ad5023690c80f3a49c8f13f8d45b8c857fbcbc8bc4a8e4d3eb4b10f4d4604fa08dce601aaf0f470216fe1b51850b4acf21b179c45070ac7b03a9
*
* this may be the same value as pubKey
*/
private String scriptHex;
/**
* Reward
*
* extracted from /src/chainparams.cpp class
* CreateGenesisBlock(..., [reward] * COIN)
*
* ie LTC = 50, BTC = 50, DOGE = 88
*/
private int reward;
/**
* Genesis Creation Version
*/
private int genesisCreationVersion;
/**
* Genesis Block Version
*/
private long genesisBlockVersion;
/**
* Genesis Time
*
* extracted from /src/chainparams.cpp class
* CreateGenesisBlock(nTime, ...)
*
* ie LTC = 1317972665
*/
private long genesisTime;
/**
* Difficulty Target
*
* extracted from /src/chainparams.cpp class
* CreateGenesisBlock(genesisTime, nonce, difficultyTarget, 1, reward * COIN);
*
* convert from hex to decimal
*
* ie LTC = 0x1e0ffff0 = 504365040
*/
private long difficultyTarget;
/**
* Merkle Hex
*/
private String merkleHex;
/**
* Nonce
*
* extracted from /src/chainparams.cpp class
* CreateGenesisBlock(genesisTime, nonce, difficultyTarget, 1, reward * COIN);
*
* ie LTC = 2084524493
*/
private long nonce;
public int getTargetTimespan() {
return targetTimespan;
}
public int getTargetSpacing() {
return targetSpacing;
}
public long getPacketMagic() {
return packetMagic;
}
public int getPort() {
return port;
}
public int getAddressHeader() {
return addressHeader;
}
public int getP2shHeader() {
return p2shHeader;
}
public String getSegwitAddressHrp() {
return segwitAddressHrp;
}
public int getDumpedPrivateKeyHeader() {
return dumpedPrivateKeyHeader;
}
public int getSubsidyDecreaseBlockCount() {
return subsidyDecreaseBlockCount;
}
public String getExpectedGenesisHash() {
return expectedGenesisHash;
}
public String getPubKey() {
return pubKey;
}
public String[] getDnsSeeds() {
return dnsSeeds;
}
public int getBip32HeaderP2PKHpub() {
return bip32HeaderP2PKHpub;
}
public int getBip32HeaderP2PKHpriv() {
return bip32HeaderP2PKHpriv;
}
public int getAddressHeaderTestnet() {
return addressHeaderTestnet;
}
public int getBip32HeaderP2PKHpubTestnet() {
return bip32HeaderP2PKHpubTestnet;
}
public int getBip32HeaderP2PKHprivTestnet() {
return bip32HeaderP2PKHprivTestnet;
}
public String getId() {
return this.id;
}
public int getMajorityEnforceBlockUpgrade() {
return this.majorityEnforceBlockUpgrade;
}
public int getMajorityRejectBlockOutdated() {
return this.majorityRejectBlockOutdated;
}
public int getMajorityWindow() {
return this.majorityWindow;
}
public String getCode() {
return this.code;
}
public String getmCode() {
return this.mCode;
}
public String getBaseCode() {
return this.baseCode;
}
public int getMinNonDustOutput() {
return this.minNonDustOutput;
}
public String getUriScheme() {
return this.uriScheme;
}
public int getProtocolVersionMinimum() {
return this.protocolVersionMinimum;
}
public int getProtocolVersionCurrent() {
return this.protocolVersionCurrent;
}
public boolean isHasMaxMoney() {
return this.hasMaxMoney;
}
public long getMaxMoney() {
return this.maxMoney;
}
public String getCurrencyCode() {
return this.currencyCode;
}
public long getMinimumOrderAmount() {
return this.minimumOrderAmount;
}
public long getFeePerKb() {
return this.feePerKb;
}
public String getNetworkName() {
return this.networkName;
}
public long getFeeRequired() {
return this.feeRequired;
}
public String getExtendedPublicKey() {
return this.extendedPublicKey;
}
public long getSendAmount() {
return this.sendAmount;
}
public long getSendingFeePerByte() {
return this.sendingFeePerByte;
}
public String getReceivingAddress() {
return this.receivingAddress;
}
public String getExtendedPrivateKey() {
return this.extendedPrivateKey;
}
public ServerInfo getServerInfo() {
return this.serverInfo;
}
public String getScriptSig() {
return this.scriptSig;
}
public String getScriptHex() {
return this.scriptHex;
}
public int getReward() {
return this.reward;
}
public int getGenesisCreationVersion() {
return this.genesisCreationVersion;
}
public long getGenesisBlockVersion() {
return this.genesisBlockVersion;
}
public long getGenesisTime() {
return this.genesisTime;
}
public long getDifficultyTarget() {
return this.difficultyTarget;
}
public String getMerkleHex() {
return this.merkleHex;
}
public long getNonce() {
return this.nonce;
}
@Override
public String toString() {
return "BitcoinyTBDRequest{" +
"targetTimespan=" + targetTimespan +
", targetSpacing=" + targetSpacing +
", packetMagic=" + packetMagic +
", port=" + port +
", addressHeader=" + addressHeader +
", p2shHeader=" + p2shHeader +
", segwitAddressHrp='" + segwitAddressHrp + '\'' +
", dumpedPrivateKeyHeader=" + dumpedPrivateKeyHeader +
", subsidyDecreaseBlockCount=" + subsidyDecreaseBlockCount +
", expectedGenesisHash='" + expectedGenesisHash + '\'' +
", pubKey='" + pubKey + '\'' +
", dnsSeeds=" + Arrays.toString(dnsSeeds) +
", bip32HeaderP2PKHpub=" + bip32HeaderP2PKHpub +
", bip32HeaderP2PKHpriv=" + bip32HeaderP2PKHpriv +
", addressHeaderTestnet=" + addressHeaderTestnet +
", bip32HeaderP2PKHpubTestnet=" + bip32HeaderP2PKHpubTestnet +
", bip32HeaderP2PKHprivTestnet=" + bip32HeaderP2PKHprivTestnet +
", id='" + id + '\'' +
", majorityEnforceBlockUpgrade=" + majorityEnforceBlockUpgrade +
", majorityRejectBlockOutdated=" + majorityRejectBlockOutdated +
", majorityWindow=" + majorityWindow +
", code='" + code + '\'' +
", mCode='" + mCode + '\'' +
", baseCode='" + baseCode + '\'' +
", minNonDustOutput=" + minNonDustOutput +
", uriScheme='" + uriScheme + '\'' +
", protocolVersionMinimum=" + protocolVersionMinimum +
", protocolVersionCurrent=" + protocolVersionCurrent +
", hasMaxMoney=" + hasMaxMoney +
", maxMoney=" + maxMoney +
", currencyCode='" + currencyCode + '\'' +
", minimumOrderAmount=" + minimumOrderAmount +
", feePerKb=" + feePerKb +
", networkName='" + networkName + '\'' +
", feeRequired=" + feeRequired +
", extendedPublicKey='" + extendedPublicKey + '\'' +
", sendAmount=" + sendAmount +
", sendingFeePerByte=" + sendingFeePerByte +
", receivingAddress='" + receivingAddress + '\'' +
", extendedPrivateKey='" + extendedPrivateKey + '\'' +
", serverInfo=" + serverInfo +
", scriptSig='" + scriptSig + '\'' +
", scriptHex='" + scriptHex + '\'' +
", reward=" + reward +
", genesisCreationVersion=" + genesisCreationVersion +
", genesisBlockVersion=" + genesisBlockVersion +
", genesisTime=" + genesisTime +
", difficultyTarget=" + difficultyTarget +
", merkleHex='" + merkleHex + '\'' +
", nonce=" + nonce +
'}';
}
}

View File

@ -1,68 +0,0 @@
package org.qortal.api.model.crosschain;
import io.swagger.v3.oas.annotations.media.Schema;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import java.util.List;
@XmlAccessorType(XmlAccessType.FIELD)
public class TradeBotRespondRequests {
@Schema(description = "Foreign blockchain private key, e.g. BIP32 'm' key for Bitcoin/Litecoin starting with 'xprv'",
example = "xprv___________________________________________________________________________________________________________")
public String foreignKey;
@Schema(description = "List of address matches")
@XmlElement(name = "addresses")
public List<String> addresses;
@Schema(description = "Qortal address for receiving QORT from AT", example = "Qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq")
public String receivingAddress;
public TradeBotRespondRequests() {
}
public TradeBotRespondRequests(String foreignKey, List<String> addresses, String receivingAddress) {
this.foreignKey = foreignKey;
this.addresses = addresses;
this.receivingAddress = receivingAddress;
}
@Schema(description = "Address Match")
// All properties to be converted to JSON via JAX-RS
@XmlAccessorType(XmlAccessType.FIELD)
public static class AddressMatch {
@Schema(description = "AT Address")
public String atAddress;
@Schema(description = "Receiving Address")
public String receivingAddress;
// For JAX-RS
protected AddressMatch() {
}
public AddressMatch(String atAddress, String receivingAddress) {
this.atAddress = atAddress;
this.receivingAddress = receivingAddress;
}
@Override
public String toString() {
return "AddressMatch{" +
"atAddress='" + atAddress + '\'' +
", receivingAddress='" + receivingAddress + '\'' +
'}';
}
}
@Override
public String toString() {
return "TradeBotRespondRequests{" +
"foreignKey='" + foreignKey + '\'' +
", addresses=" + addresses +
'}';
}
}

View File

@ -142,20 +142,10 @@ public class DevProxyServerResource {
} }
} }
String lang = request.getParameter("lang");
if (lang == null || lang.isBlank()) {
lang = "en"; // fallback
}
String theme = request.getParameter("theme");
if (theme == null || theme.isBlank()) {
theme = "light";
}
// Parse and modify output if needed // Parse and modify output if needed
if (HTMLParser.isHtmlFile(filename)) { if (HTMLParser.isHtmlFile(filename)) {
// HTML file - needs to be parsed // HTML file - needs to be parsed
HTMLParser htmlParser = new HTMLParser("", inPath, "", false, data, "proxy", Service.APP, null, theme , true, lang); HTMLParser htmlParser = new HTMLParser("", inPath, "", false, data, "proxy", Service.APP, null, "light", true);
htmlParser.addAdditionalHeaderTags(); htmlParser.addAdditionalHeaderTags();
response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' ws:; font-src 'self' data:;"); response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' ws:; font-src 'self' data:;");
response.setContentType(con.getContentType()); response.setContentType(con.getContentType());

View File

@ -20,7 +20,9 @@ import org.qortal.asset.Asset;
import org.qortal.controller.LiteNode; import org.qortal.controller.LiteNode;
import org.qortal.controller.OnlineAccountsManager; import org.qortal.controller.OnlineAccountsManager;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
import org.qortal.data.account.*; import org.qortal.data.account.AccountData;
import org.qortal.data.account.AccountPenaltyData;
import org.qortal.data.account.RewardShareData;
import org.qortal.data.network.OnlineAccountData; import org.qortal.data.network.OnlineAccountData;
import org.qortal.data.network.OnlineAccountLevel; import org.qortal.data.network.OnlineAccountLevel;
import org.qortal.data.transaction.PublicizeTransactionData; import org.qortal.data.transaction.PublicizeTransactionData;
@ -50,7 +52,6 @@ import java.math.BigDecimal;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Comparator; import java.util.Comparator;
import java.util.List; import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors; import java.util.stream.Collectors;
@Path("/addresses") @Path("/addresses")
@ -326,8 +327,11 @@ public class AddressesResource {
) )
} }
) )
@ApiErrors({ApiError.INVALID_PUBLIC_KEY, ApiError.REPOSITORY_ISSUE}) @ApiErrors({ApiError.INVALID_PUBLIC_KEY, ApiError.NON_PRODUCTION, ApiError.REPOSITORY_ISSUE})
public String fromPublicKey(@PathParam("publickey") String publicKey58) { public String fromPublicKey(@PathParam("publickey") String publicKey58) {
if (Settings.getInstance().isApiRestricted())
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION);
// Decode public key // Decode public key
byte[] publicKey; byte[] publicKey;
try { try {
@ -626,160 +630,4 @@ public class AddressesResource {
} }
} }
@GET
@Path("/sponsorship/{address}")
@Operation(
summary = "Returns sponsorship statistics for an account",
description = "Returns sponsorship statistics for an account, excluding the recipients that get real reward shares",
responses = {
@ApiResponse(
description = "the statistics",
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = SponsorshipReport.class))
)
}
)
@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.ADDRESS_UNKNOWN, ApiError.REPOSITORY_ISSUE})
public SponsorshipReport getSponsorshipReport(
@PathParam("address") String address,
@QueryParam(("realRewardShareRecipient")) String[] realRewardShareRecipients) {
if (!Crypto.isValidAddress(address))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
try (final Repository repository = RepositoryManager.getRepository()) {
SponsorshipReport report = repository.getAccountRepository().getSponsorshipReport(address, realRewardShareRecipients);
// Not found?
if (report == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.ADDRESS_UNKNOWN);
return report;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/sponsorship/{address}/sponsor")
@Operation(
summary = "Returns sponsorship statistics for an account's sponsor",
description = "Returns sponsorship statistics for an account's sponsor, excluding the recipients that get real reward shares",
responses = {
@ApiResponse(
description = "the statistics",
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = SponsorshipReport.class))
)
}
)
@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.ADDRESS_UNKNOWN, ApiError.REPOSITORY_ISSUE})
public SponsorshipReport getSponsorshipReportForSponsor(
@PathParam("address") String address,
@QueryParam("realRewardShareRecipient") String[] realRewardShareRecipients) {
if (!Crypto.isValidAddress(address))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
try (final Repository repository = RepositoryManager.getRepository()) {
// get sponsor
Optional<String> sponsor = repository.getAccountRepository().getSponsor(address);
// if there is not sponsor, throw error
if(sponsor.isEmpty()) throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.ADDRESS_UNKNOWN);
// get report for sponsor
SponsorshipReport report = repository.getAccountRepository().getSponsorshipReport(sponsor.get(), realRewardShareRecipients);
// Not found?
if (report == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.ADDRESS_UNKNOWN);
return report;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/mintership/{address}")
@Operation(
summary = "Returns mintership statistics for an account",
description = "Returns mintership statistics for an account",
responses = {
@ApiResponse(
description = "the statistics",
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = MintershipReport.class))
)
}
)
@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.ADDRESS_UNKNOWN, ApiError.REPOSITORY_ISSUE})
public MintershipReport getMintershipReport(@PathParam("address") String address,
@QueryParam("realRewardShareRecipient") String[] realRewardShareRecipients ) {
if (!Crypto.isValidAddress(address))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
try (final Repository repository = RepositoryManager.getRepository()) {
// get sponsorship report for minter, fetch a list of one minter
SponsorshipReport report = repository.getAccountRepository().getMintershipReport(address, account -> List.of(account));
// Not found?
if (report == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.ADDRESS_UNKNOWN);
// since the report is for one minter, must get sponsee count separately
int sponseeCount = repository.getAccountRepository().getSponseeAddresses(address, realRewardShareRecipients).size();
// since the report is for one minter, must get the first name from a array of names that should be size 1
String name = report.getNames().length > 0 ? report.getNames()[0] : null;
// transform sponsorship report to mintership report
MintershipReport mintershipReport
= new MintershipReport(
report.getAddress(),
report.getLevel(),
report.getBlocksMinted(),
report.getAdjustments(),
report.getPenalties(),
report.isTransfer(),
name,
sponseeCount,
report.getAvgBalance(),
report.getArbitraryCount(),
report.getTransferAssetCount(),
report.getTransferPrivsCount(),
report.getSellCount(),
report.getSellAmount(),
report.getBuyCount(),
report.getBuyAmount()
);
return mintershipReport;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/levels/{minLevel}")
@Operation(
summary = "Return accounts with levels greater than or equal to input",
responses = {
@ApiResponse(
description = "online accounts",
content = @Content(mediaType = MediaType.APPLICATION_JSON, array = @ArraySchema(schema = @Schema(implementation = AddressLevelPairing.class)))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<AddressLevelPairing> getAddressLevelPairings(@PathParam("minLevel") int minLevel) {
try (final Repository repository = RepositoryManager.getRepository()) {
// get the level address pairings
List<AddressLevelPairing> pairings = repository.getAccountRepository().getAddressLevelPairings(minLevel);
return pairings;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
} }

View File

@ -3,7 +3,6 @@ package org.qortal.api.resource;
import com.google.common.primitives.Bytes; import com.google.common.primitives.Bytes;
import com.j256.simplemagic.ContentInfo; import com.j256.simplemagic.ContentInfo;
import com.j256.simplemagic.ContentInfoUtil; import com.j256.simplemagic.ContentInfoUtil;
import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter; import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.ArraySchema; import io.swagger.v3.oas.annotations.media.ArraySchema;
@ -13,7 +12,6 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody;
import io.swagger.v3.oas.annotations.responses.ApiResponse; import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.security.SecurityRequirement; import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag; import io.swagger.v3.oas.annotations.tags.Tag;
import org.apache.commons.io.FileUtils; import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.ArrayUtils; import org.apache.commons.lang3.ArrayUtils;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
@ -35,13 +33,9 @@ import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
import org.qortal.controller.arbitrary.ArbitraryMetadataManager; import org.qortal.controller.arbitrary.ArbitraryMetadataManager;
import org.qortal.data.account.AccountData; import org.qortal.data.account.AccountData;
import org.qortal.data.arbitrary.ArbitraryCategoryInfo; import org.qortal.data.arbitrary.ArbitraryCategoryInfo;
import org.qortal.data.arbitrary.ArbitraryDataIndexDetail;
import org.qortal.data.arbitrary.ArbitraryDataIndexScoreKey;
import org.qortal.data.arbitrary.ArbitraryDataIndexScorecard;
import org.qortal.data.arbitrary.ArbitraryResourceData; import org.qortal.data.arbitrary.ArbitraryResourceData;
import org.qortal.data.arbitrary.ArbitraryResourceMetadata; import org.qortal.data.arbitrary.ArbitraryResourceMetadata;
import org.qortal.data.arbitrary.ArbitraryResourceStatus; import org.qortal.data.arbitrary.ArbitraryResourceStatus;
import org.qortal.data.arbitrary.IndexCache;
import org.qortal.data.naming.NameData; import org.qortal.data.naming.NameData;
import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
@ -65,36 +59,18 @@ import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.*; import javax.ws.rs.*;
import javax.ws.rs.core.Context; import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType; import javax.ws.rs.core.MediaType;
import java.io.BufferedWriter; import java.io.BufferedWriter;
import java.io.File; import java.io.File;
import java.io.FileWriter; import java.io.FileWriter;
import java.io.IOException; import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.FileNameMap; import java.net.FileNameMap;
import java.net.URLConnection; import java.net.URLConnection;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Arrays; import java.util.Arrays;
import java.util.Comparator;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.Objects; import java.util.Objects;
import java.util.stream.Collectors;
import java.util.zip.GZIPOutputStream;
import org.apache.tika.Tika;
import org.apache.tika.mime.MimeTypeException;
import org.apache.tika.mime.MimeTypes;
import javax.ws.rs.core.Response;
import org.glassfish.jersey.media.multipart.FormDataParam;
import static org.qortal.api.ApiError.REPOSITORY_ISSUE;
@Path("/arbitrary") @Path("/arbitrary")
@Tag(name = "Arbitrary") @Tag(name = "Arbitrary")
@ -196,7 +172,6 @@ public class ArbitraryResource {
@Parameter(description = "Name (searches name field only)") @QueryParam("name") List<String> names, @Parameter(description = "Name (searches name field only)") @QueryParam("name") List<String> names,
@Parameter(description = "Title (searches title metadata field only)") @QueryParam("title") String title, @Parameter(description = "Title (searches title metadata field only)") @QueryParam("title") String title,
@Parameter(description = "Description (searches description metadata field only)") @QueryParam("description") String description, @Parameter(description = "Description (searches description metadata field only)") @QueryParam("description") String description,
@Parameter(description = "Keyword (searches description metadata field by keywords)") @QueryParam("keywords") List<String> keywords,
@Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly, @Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly,
@Parameter(description = "Exact match names only (if true, partial name matches are excluded)") @QueryParam("exactmatchnames") Boolean exactMatchNamesOnly, @Parameter(description = "Exact match names only (if true, partial name matches are excluded)") @QueryParam("exactmatchnames") Boolean exactMatchNamesOnly,
@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource, @Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
@ -237,7 +212,7 @@ public class ArbitraryResource {
} }
List<ArbitraryResourceData> resources = repository.getArbitraryRepository() List<ArbitraryResourceData> resources = repository.getArbitraryRepository()
.searchArbitraryResources(service, query, identifier, names, title, description, keywords, usePrefixOnly, .searchArbitraryResources(service, query, identifier, names, title, description, usePrefixOnly,
exactMatchNames, defaultRes, mode, minLevel, followedOnly, excludeBlocked, includeMetadata, includeStatus, exactMatchNames, defaultRes, mode, minLevel, followedOnly, excludeBlocked, includeMetadata, includeStatus,
before, after, limit, offset, reverse); before, after, limit, offset, reverse);
@ -252,49 +227,6 @@ public class ArbitraryResource {
} }
} }
@GET
@Path("/resources/searchsimple")
@Operation(
summary = "Search arbitrary resources available on chain, optionally filtered by service.",
responses = {
@ApiResponse(
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = ArbitraryResourceData.class))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<ArbitraryResourceData> searchResourcesSimple(
@QueryParam("service") Service service,
@Parameter(description = "Identifier (searches identifier field only)") @QueryParam("identifier") String identifier,
@Parameter(description = "Name (searches name field only)") @QueryParam("name") List<String> names,
@Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly,
@Parameter(description = "Case insensitive (ignore leter case on search)") @QueryParam("caseInsensitive") Boolean caseInsensitive,
@Parameter(description = "Creation date before timestamp") @QueryParam("before") Long before,
@Parameter(description = "Creation date after timestamp") @QueryParam("after") Long after,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
try (final Repository repository = RepositoryManager.getRepository()) {
boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly);
boolean ignoreCase = Boolean.TRUE.equals(caseInsensitive);
List<ArbitraryResourceData> resources = repository.getArbitraryRepository()
.searchArbitraryResourcesSimple(service, identifier, names, usePrefixOnly,
before, after, limit, offset, reverse, ignoreCase);
if (resources == null) {
return new ArrayList<>();
}
return resources;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET @GET
@Path("/resource/status/{service}/{name}") @Path("/resource/status/{service}/{name}")
@Operation( @Operation(
@ -703,20 +635,20 @@ public class ArbitraryResource {
) )
} }
) )
public void get(@PathParam("service") Service service, public HttpServletResponse get(@PathParam("service") Service service,
@PathParam("name") String name, @PathParam("name") String name,
@QueryParam("filepath") String filepath, @QueryParam("filepath") String filepath,
@QueryParam("encoding") String encoding, @QueryParam("encoding") String encoding,
@QueryParam("rebuild") boolean rebuild, @QueryParam("rebuild") boolean rebuild,
@QueryParam("async") boolean async, @QueryParam("async") boolean async,
@QueryParam("attempts") Integer attempts, @QueryParam("attachment") boolean attachment, @QueryParam("attachmentFilename") String attachmentFilename) { @QueryParam("attempts") Integer attempts) {
// Authentication can be bypassed in the settings, for those running public QDN nodes // Authentication can be bypassed in the settings, for those running public QDN nodes
if (!Settings.getInstance().isQDNAuthBypassEnabled()) { if (!Settings.getInstance().isQDNAuthBypassEnabled()) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
} }
this.download(service, name, null, filepath, encoding, rebuild, async, attempts, attachment, attachmentFilename); return this.download(service, name, null, filepath, encoding, rebuild, async, attempts);
} }
@GET @GET
@ -736,21 +668,21 @@ public class ArbitraryResource {
) )
} }
) )
public void get(@PathParam("service") Service service, public HttpServletResponse get(@PathParam("service") Service service,
@PathParam("name") String name, @PathParam("name") String name,
@PathParam("identifier") String identifier, @PathParam("identifier") String identifier,
@QueryParam("filepath") String filepath, @QueryParam("filepath") String filepath,
@QueryParam("encoding") String encoding, @QueryParam("encoding") String encoding,
@QueryParam("rebuild") boolean rebuild, @QueryParam("rebuild") boolean rebuild,
@QueryParam("async") boolean async, @QueryParam("async") boolean async,
@QueryParam("attempts") Integer attempts, @QueryParam("attachment") boolean attachment, @QueryParam("attachmentFilename") String attachmentFilename) { @QueryParam("attempts") Integer attempts) {
// Authentication can be bypassed in the settings, for those running public QDN nodes // Authentication can be bypassed in the settings, for those running public QDN nodes
if (!Settings.getInstance().isQDNAuthBypassEnabled()) { if (!Settings.getInstance().isQDNAuthBypassEnabled()) {
Security.checkApiCallAllowed(request, null); Security.checkApiCallAllowed(request, null);
} }
this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts, attachment, attachmentFilename); return this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts);
} }
@ -895,464 +827,6 @@ public class ArbitraryResource {
} }
@GET
@Path("/check/tmp")
@Produces(MediaType.TEXT_PLAIN)
@Operation(
summary = "Check if the disk has enough disk space for an upcoming upload",
responses = {
@ApiResponse(description = "OK if sufficient space", responseCode = "200"),
@ApiResponse(description = "Insufficient space", responseCode = "507") // 507 = Insufficient Storage
}
)
@SecurityRequirement(name = "apiKey")
public Response checkUploadSpace(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@QueryParam("totalSize") Long totalSize) {
Security.checkApiCallAllowed(request);
if (totalSize == null || totalSize <= 0) {
return Response.status(Response.Status.BAD_REQUEST)
.entity("Missing or invalid totalSize parameter").build();
}
File uploadDir = new File("uploads-temp");
if (!uploadDir.exists()) {
uploadDir.mkdirs(); // ensure the folder exists
}
long usableSpace = uploadDir.getUsableSpace();
long requiredSpace = (long)(((double)totalSize) * 2.2); // estimate for chunks + merge
if (usableSpace < requiredSpace) {
return Response.status(507).entity("Insufficient disk space").build();
}
return Response.ok("Sufficient disk space").build();
}
@POST
@Path("/{service}/{name}/chunk")
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Operation(
summary = "Upload a single file chunk to be later assembled into a complete arbitrary resource (no identifier)",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.MULTIPART_FORM_DATA,
schema = @Schema(
implementation = Object.class
)
)
),
responses = {
@ApiResponse(
description = "Chunk uploaded successfully",
responseCode = "200"
),
@ApiResponse(
description = "Error writing chunk",
responseCode = "500"
)
}
)
@SecurityRequirement(name = "apiKey")
public Response uploadChunkNoIdentifier(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@PathParam("service") String serviceString,
@PathParam("name") String name,
@FormDataParam("chunk") InputStream chunkStream,
@FormDataParam("index") int index) {
Security.checkApiCallAllowed(request);
try {
String safeService = Paths.get(serviceString).getFileName().toString();
String safeName = Paths.get(name).getFileName().toString();
java.nio.file.Path tempDir = Paths.get("uploads-temp", safeService, safeName);
Files.createDirectories(tempDir);
java.nio.file.Path chunkFile = tempDir.resolve("chunk_" + index);
Files.copy(chunkStream, chunkFile, StandardCopyOption.REPLACE_EXISTING);
return Response.ok("Chunk " + index + " received").build();
} catch (IOException e) {
LOGGER.error("Failed to write chunk {} for service '{}' and name '{}'", index, serviceString, name, e);
return Response.serverError().entity("Failed to write chunk: " + e.getMessage()).build();
}
}
@POST
@Path("/{service}/{name}/finalize")
@Produces(MediaType.TEXT_PLAIN)
@Operation(
summary = "Finalize a chunked upload (no identifier) and build a raw, unsigned, ARBITRARY transaction",
responses = {
@ApiResponse(
description = "raw, unsigned, ARBITRARY transaction encoded in Base58",
content = @Content(mediaType = MediaType.TEXT_PLAIN)
)
}
)
@SecurityRequirement(name = "apiKey")
public String finalizeUploadNoIdentifier(
@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@PathParam("service") String serviceString,
@PathParam("name") String name,
@QueryParam("title") String title,
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("filename") String filename,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
@QueryParam("isZip") Boolean isZip
) {
Security.checkApiCallAllowed(request);
java.nio.file.Path tempFile = null;
java.nio.file.Path tempDir = null;
java.nio.file.Path chunkDir = null;
String safeService = Paths.get(serviceString).getFileName().toString();
String safeName = Paths.get(name).getFileName().toString();
try {
chunkDir = Paths.get("uploads-temp", safeService, safeName);
if (!Files.exists(chunkDir) || !Files.isDirectory(chunkDir)) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "No chunks found for upload");
}
String safeFilename = (filename == null || filename.isBlank()) ? "qortal-" + NTP.getTime() : filename;
tempDir = Files.createTempDirectory("qortal-");
String sanitizedFilename = Paths.get(safeFilename).getFileName().toString();
tempFile = tempDir.resolve(sanitizedFilename);
try (OutputStream out = Files.newOutputStream(tempFile, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING)) {
byte[] buffer = new byte[65536];
for (java.nio.file.Path chunk : Files.list(chunkDir)
.filter(path -> path.getFileName().toString().startsWith("chunk_"))
.sorted(Comparator.comparingInt(path -> {
String name2 = path.getFileName().toString();
String numberPart = name2.substring("chunk_".length());
return Integer.parseInt(numberPart);
})).collect(Collectors.toList())) {
try (InputStream in = Files.newInputStream(chunk)) {
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
}
}
}
String detectedExtension = "";
String uploadFilename = null;
boolean extensionIsValid = false;
if (filename != null && !filename.isBlank()) {
int lastDot = filename.lastIndexOf('.');
if (lastDot > 0 && lastDot < filename.length() - 1) {
extensionIsValid = true;
uploadFilename = filename;
}
}
if (!extensionIsValid) {
Tika tika = new Tika();
String mimeType = tika.detect(tempFile.toFile());
try {
MimeTypes allTypes = MimeTypes.getDefaultMimeTypes();
org.apache.tika.mime.MimeType mime = allTypes.forName(mimeType);
detectedExtension = mime.getExtension();
} catch (MimeTypeException e) {
LOGGER.warn("Could not determine file extension for MIME type: {}", mimeType, e);
}
if (filename != null && !filename.isBlank()) {
int lastDot = filename.lastIndexOf('.');
String baseName = (lastDot > 0) ? filename.substring(0, lastDot) : filename;
uploadFilename = baseName + (detectedExtension != null ? detectedExtension : "");
} else {
uploadFilename = "qortal-" + NTP.getTime() + (detectedExtension != null ? detectedExtension : "");
}
}
Boolean isZipBoolean = false;
if (isZip != null && isZip) {
isZipBoolean = true;
}
// Call upload with `null` as identifier
return this.upload(
Service.valueOf(serviceString),
name,
null, // no identifier
tempFile.toString(),
null,
null,
isZipBoolean,
fee,
uploadFilename,
title,
description,
tags,
category,
preview
);
} catch (IOException e) {
LOGGER.error("Failed to merge chunks for service='{}', name='{}'", serviceString, name, e);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, "Failed to merge chunks: " + e.getMessage());
} finally {
if (tempDir != null) {
try {
Files.walk(tempDir)
.sorted(Comparator.reverseOrder())
.map(java.nio.file.Path::toFile)
.forEach(File::delete);
} catch (IOException e) {
LOGGER.warn("Failed to delete temp directory: {}", tempDir, e);
}
}
try {
Files.walk(chunkDir)
.sorted(Comparator.reverseOrder())
.map(java.nio.file.Path::toFile)
.forEach(File::delete);
} catch (IOException e) {
LOGGER.warn("Failed to delete chunk directory: {}", chunkDir, e);
}
}
}
@POST
@Path("/{service}/{name}/{identifier}/chunk")
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Operation(
summary = "Upload a single file chunk to be later assembled into a complete arbitrary resource",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.MULTIPART_FORM_DATA,
schema = @Schema(
implementation = Object.class
)
)
),
responses = {
@ApiResponse(
description = "Chunk uploaded successfully",
responseCode = "200"
),
@ApiResponse(
description = "Error writing chunk",
responseCode = "500"
)
}
)
@SecurityRequirement(name = "apiKey")
public Response uploadChunk(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@PathParam("service") String serviceString,
@PathParam("name") String name,
@PathParam("identifier") String identifier,
@FormDataParam("chunk") InputStream chunkStream,
@FormDataParam("index") int index) {
Security.checkApiCallAllowed(request);
try {
String safeService = Paths.get(serviceString).getFileName().toString();
String safeName = Paths.get(name).getFileName().toString();
String safeIdentifier = Paths.get(identifier).getFileName().toString();
java.nio.file.Path tempDir = Paths.get("uploads-temp", safeService, safeName, safeIdentifier);
Files.createDirectories(tempDir);
java.nio.file.Path chunkFile = tempDir.resolve("chunk_" + index);
Files.copy(chunkStream, chunkFile, StandardCopyOption.REPLACE_EXISTING);
return Response.ok("Chunk " + index + " received").build();
} catch (IOException e) {
LOGGER.error("Failed to write chunk {} for service='{}', name='{}', identifier='{}'", index, serviceString, name, identifier, e);
return Response.serverError().entity("Failed to write chunk: " + e.getMessage()).build();
}
}
@POST
@Path("/{service}/{name}/{identifier}/finalize")
@Produces(MediaType.TEXT_PLAIN)
@Operation(
summary = "Finalize a chunked upload and build a raw, unsigned, ARBITRARY transaction",
responses = {
@ApiResponse(
description = "raw, unsigned, ARBITRARY transaction encoded in Base58",
content = @Content(mediaType = MediaType.TEXT_PLAIN)
)
}
)
@SecurityRequirement(name = "apiKey")
public String finalizeUpload(
@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@PathParam("service") String serviceString,
@PathParam("name") String name,
@PathParam("identifier") String identifier,
@QueryParam("title") String title,
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("filename") String filename,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
@QueryParam("isZip") Boolean isZip
) {
Security.checkApiCallAllowed(request);
java.nio.file.Path tempFile = null;
java.nio.file.Path tempDir = null;
java.nio.file.Path chunkDir = null;
try {
String safeService = Paths.get(serviceString).getFileName().toString();
String safeName = Paths.get(name).getFileName().toString();
String safeIdentifier = Paths.get(identifier).getFileName().toString();
java.nio.file.Path baseUploadsDir = Paths.get("uploads-temp"); // relative to Qortal working dir
chunkDir = baseUploadsDir.resolve(safeService).resolve(safeName).resolve(safeIdentifier);
if (!Files.exists(chunkDir) || !Files.isDirectory(chunkDir)) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "No chunks found for upload");
}
// Step 1: Determine a safe filename for disk temp file (regardless of extension correctness)
String safeFilename = filename;
if (filename == null || filename.isBlank()) {
safeFilename = "qortal-" + NTP.getTime();
}
tempDir = Files.createTempDirectory("qortal-");
String sanitizedFilename = Paths.get(safeFilename).getFileName().toString();
tempFile = tempDir.resolve(sanitizedFilename);
// Step 2: Merge chunks
try (OutputStream out = Files.newOutputStream(tempFile, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING)) {
byte[] buffer = new byte[65536];
for (java.nio.file.Path chunk : Files.list(chunkDir)
.filter(path -> path.getFileName().toString().startsWith("chunk_"))
.sorted(Comparator.comparingInt(path -> {
String name2 = path.getFileName().toString();
String numberPart = name2.substring("chunk_".length());
return Integer.parseInt(numberPart);
})).collect(Collectors.toList())) {
try (InputStream in = Files.newInputStream(chunk)) {
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
}
}
}
// Step 3: Determine correct extension
String detectedExtension = "";
String uploadFilename = null;
boolean extensionIsValid = false;
if (filename != null && !filename.isBlank()) {
int lastDot = filename.lastIndexOf('.');
if (lastDot > 0 && lastDot < filename.length() - 1) {
extensionIsValid = true;
uploadFilename = filename;
}
}
if (!extensionIsValid) {
Tika tika = new Tika();
String mimeType = tika.detect(tempFile.toFile());
try {
MimeTypes allTypes = MimeTypes.getDefaultMimeTypes();
org.apache.tika.mime.MimeType mime = allTypes.forName(mimeType);
detectedExtension = mime.getExtension();
} catch (MimeTypeException e) {
LOGGER.warn("Could not determine file extension for MIME type: {}", mimeType, e);
}
if (filename != null && !filename.isBlank()) {
int lastDot = filename.lastIndexOf('.');
String baseName = (lastDot > 0) ? filename.substring(0, lastDot) : filename;
uploadFilename = baseName + (detectedExtension != null ? detectedExtension : "");
} else {
uploadFilename = "qortal-" + NTP.getTime() + (detectedExtension != null ? detectedExtension : "");
}
}
Boolean isZipBoolean = false;
if (isZip != null && isZip) {
isZipBoolean = true;
}
return this.upload(
Service.valueOf(serviceString),
name,
identifier,
tempFile.toString(),
null,
null,
isZipBoolean,
fee,
uploadFilename,
title,
description,
tags,
category,
preview
);
} catch (IOException e) {
LOGGER.error("Unexpected error in finalizeUpload for service='{}', name='{}', name='{}'", serviceString, name, identifier, e);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, "Failed to merge chunks: " + e.getMessage());
} finally {
if (tempDir != null) {
try {
Files.walk(tempDir)
.sorted(Comparator.reverseOrder())
.map(java.nio.file.Path::toFile)
.forEach(File::delete);
} catch (IOException e) {
LOGGER.warn("Failed to delete temp directory: {}", tempDir, e);
}
}
try {
Files.walk(chunkDir)
.sorted(Comparator.reverseOrder())
.map(java.nio.file.Path::toFile)
.forEach(File::delete);
} catch (IOException e) {
LOGGER.warn("Failed to delete chunk directory: {}", chunkDir, e);
}
}
}
// Upload base64-encoded data // Upload base64-encoded data
@ -1668,90 +1142,6 @@ public String finalizeUpload(
} }
} }
@GET
@Path("/indices")
@Operation(
summary = "Find matching arbitrary resource indices",
description = "",
responses = {
@ApiResponse(
description = "indices",
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = ArbitraryDataIndexScorecard.class
)
)
)
)
}
)
public List<ArbitraryDataIndexScorecard> searchIndices(@QueryParam("terms") String[] terms) {
List<ArbitraryDataIndexDetail> indices = new ArrayList<>();
// get index details for each term
for( String term : terms ) {
List<ArbitraryDataIndexDetail> details = IndexCache.getInstance().getIndicesByTerm().get(term);
if( details != null ) {
indices.addAll(details);
}
}
// sum up the scores for each index with identical attributes
Map<ArbitraryDataIndexScoreKey, Double> scoreForKey
= indices.stream()
.collect(
Collectors.groupingBy(
index -> new ArbitraryDataIndexScoreKey(index.name, index.category, index.link),
Collectors.summingDouble(detail -> 1.0 / detail.rank)
)
);
// create scorecards for each index group and put them in descending order by score
List<ArbitraryDataIndexScorecard> scorecards
= scoreForKey.entrySet().stream().map(
entry
->
new ArbitraryDataIndexScorecard(
entry.getValue(),
entry.getKey().name,
entry.getKey().category,
entry.getKey().link)
)
.sorted(Comparator.comparingDouble(ArbitraryDataIndexScorecard::getScore).reversed())
.collect(Collectors.toList());
return scorecards;
}
@GET
@Path("/indices/{name}/{idPrefix}")
@Operation(
summary = "Find matching arbitrary resource indices for a registered name and identifier prefix",
description = "",
responses = {
@ApiResponse(
description = "indices",
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = ArbitraryDataIndexDetail.class
)
)
)
)
}
)
public List<ArbitraryDataIndexDetail> searchIndicesByName(@PathParam("name") String name, @PathParam("idPrefix") String idPrefix) {
return
IndexCache.getInstance().getIndicesByIssuer()
.getOrDefault(name, new ArrayList<>(0)).stream()
.filter( indexDetail -> indexDetail.indexIdentifer.startsWith(idPrefix))
.collect(Collectors.toList());
}
// Shared methods // Shared methods
@ -1818,7 +1208,7 @@ public String finalizeUpload(
if (path == null) { if (path == null) {
// See if we have a string instead // See if we have a string instead
if (string != null) { if (string != null) {
if (filename == null || filename.isBlank()) { if (filename == null) {
// Use current time as filename // Use current time as filename
filename = String.format("qortal-%d", NTP.getTime()); filename = String.format("qortal-%d", NTP.getTime());
} }
@ -1833,7 +1223,7 @@ public String finalizeUpload(
} }
// ... or base64 encoded raw data // ... or base64 encoded raw data
else if (base64 != null) { else if (base64 != null) {
if (filename == null || filename.isBlank()) { if (filename == null) {
// Use current time as filename // Use current time as filename
filename = String.format("qortal-%d", NTP.getTime()); filename = String.format("qortal-%d", NTP.getTime());
} }
@ -1884,7 +1274,6 @@ public String finalizeUpload(
); );
transactionBuilder.build(); transactionBuilder.build();
// Don't compute nonce - this is done by the client (or via POST /arbitrary/compute) // Don't compute nonce - this is done by the client (or via POST /arbitrary/compute)
ArbitraryTransactionData transactionData = transactionBuilder.getArbitraryTransactionData(); ArbitraryTransactionData transactionData = transactionBuilder.getArbitraryTransactionData();
return Base58.encode(ArbitraryTransactionTransformer.toBytes(transactionData)); return Base58.encode(ArbitraryTransactionTransformer.toBytes(transactionData));
@ -1900,7 +1289,8 @@ public String finalizeUpload(
} }
} }
private void download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts, boolean attachment, String attachmentFilename) { private HttpServletResponse download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts) {
try { try {
ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
@ -1913,7 +1303,8 @@ public String finalizeUpload(
if (async) { if (async) {
// Asynchronous // Asynchronous
arbitraryDataReader.loadAsynchronously(false, 1); arbitraryDataReader.loadAsynchronously(false, 1);
} else { }
else {
// Synchronous // Synchronous
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
attempts++; attempts++;
@ -1923,10 +1314,12 @@ public String finalizeUpload(
break; break;
} catch (MissingDataException e) { } catch (MissingDataException e) {
if (attempts > maxAttempts) { if (attempts > maxAttempts) {
// Give up after 5 attempts
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Data unavailable. Please try again later."); throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Data unavailable. Please try again later.");
} }
} }
} }
Thread.sleep(3000L);
} }
} }
@ -1942,171 +1335,68 @@ public String finalizeUpload(
if (files != null && files.length == 1) { if (files != null && files.length == 1) {
// This is a single file resource // This is a single file resource
filepath = files[0]; filepath = files[0];
} else { }
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "filepath is required for resources containing more than one file"); else {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA,
"filepath is required for resources containing more than one file");
} }
} }
java.nio.file.Path path = Paths.get(outputPath.toString(), filepath); java.nio.file.Path path = Paths.get(outputPath.toString(), filepath);
if (!Files.exists(path)) { if (!Files.exists(path)) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "No file exists at filepath: " + filepath); String message = String.format("No file exists at filepath: %s", filepath);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, message);
} }
if (attachment) { byte[] data;
String rawFilename; int fileSize = (int)path.toFile().length();
int length = fileSize;
if (attachmentFilename != null && !attachmentFilename.isEmpty()) { // Parse "Range" header
// 1. Sanitize first Integer rangeStart = null;
String safeAttachmentFilename = attachmentFilename.replaceAll("[\\\\/:*?\"<>|]", "_"); Integer rangeEnd = null;
// 2. Check for a valid extension (35 alphanumeric chars)
if (!safeAttachmentFilename.matches(".*\\.[a-zA-Z0-9]{2,5}$")) {
safeAttachmentFilename += ".bin";
}
rawFilename = safeAttachmentFilename;
} else {
// Fallback if no filename is provided
String baseFilename = (identifier != null && !identifier.isEmpty())
? name + "-" + identifier
: name;
rawFilename = baseFilename.replaceAll("[\\\\/:*?\"<>|]", "_") + ".bin";
}
// Optional: trim length
rawFilename = rawFilename.length() > 100 ? rawFilename.substring(0, 100) : rawFilename;
// 3. Set Content-Disposition header
response.setHeader("Content-Disposition", "attachment; filename=\"" + rawFilename + "\"");
}
// Determine the total size of the requested file
long fileSize = Files.size(path);
String mimeType = context.getMimeType(path.toString());
// Attempt to read the "Range" header from the request to support partial content delivery (e.g., for video streaming or resumable downloads)
String range = request.getHeader("Range"); String range = request.getHeader("Range");
if (range != null) {
long rangeStart = 0; range = range.replace("bytes=", "");
long rangeEnd = fileSize - 1; String[] parts = range.split("-");
boolean isPartial = false; rangeStart = (parts != null && parts.length > 0) ? Integer.parseInt(parts[0]) : null;
rangeEnd = (parts != null && parts.length > 1) ? Integer.parseInt(parts[1]) : fileSize;
// If a Range header is present and no base64 encoding is requested, parse the range values
if (range != null && encoding == null) {
range = range.replace("bytes=", ""); // Remove the "bytes=" prefix
String[] parts = range.split("-"); // Split the range into start and end
// Parse range start
if (parts.length > 0 && !parts[0].isEmpty()) {
rangeStart = Long.parseLong(parts[0]);
}
// Parse range end, if present
if (parts.length > 1 && !parts[1].isEmpty()) {
rangeEnd = Long.parseLong(parts[1]);
}
isPartial = true; // Indicate that this is a partial content request
} }
// Calculate how many bytes should be sent in the response if (rangeStart != null && rangeEnd != null) {
long contentLength = rangeEnd - rangeStart + 1; // We have a range, so update the requested length
length = rangeEnd - rangeStart;
// Inform the client that byte ranges are supported
response.setHeader("Accept-Ranges", "bytes");
if (isPartial) {
// If partial content was requested, return 206 Partial Content with appropriate headers
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
response.setHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd, fileSize));
} else {
// Otherwise, return the entire file with status 200 OK
response.setStatus(HttpServletResponse.SC_OK);
} }
// Initialize output streams for writing the file to the response if (length < fileSize && encoding == null) {
OutputStream rawOut = null; // Partial content requested, and not encoding the data
OutputStream base64Out = null; response.setStatus(206);
OutputStream gzipOut = null; response.addHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd-1, fileSize));
data = FilesystemUtils.readFromFile(path.toString(), rangeStart, length);
try { }
rawOut = response.getOutputStream(); else {
// Full content requested (or encoded data)
if (encoding != null && "base64".equalsIgnoreCase(encoding)) { response.setStatus(200);
// If base64 encoding is requested, override content type data = Files.readAllBytes(path); // TODO: limit file size that can be read into memory
response.setContentType("text/plain");
// Check if the client accepts gzip encoding
String acceptEncoding = request.getHeader("Accept-Encoding");
boolean wantsGzip = acceptEncoding != null && acceptEncoding.contains("gzip");
if (wantsGzip) {
// Wrap output in GZIP and Base64 streams if gzip is accepted
response.setHeader("Content-Encoding", "gzip");
gzipOut = new GZIPOutputStream(rawOut);
base64Out = java.util.Base64.getEncoder().wrap(gzipOut);
} else {
// Wrap output in Base64 only
base64Out = java.util.Base64.getEncoder().wrap(rawOut);
}
rawOut = base64Out; // Use the wrapped stream for writing
} else {
// For raw binary output, set the content type and length
response.setContentType(mimeType != null ? mimeType : "application/octet-stream");
response.setContentLength((int) contentLength);
}
// Stream file content
try (InputStream inputStream = Files.newInputStream(path)) {
if (rangeStart > 0) {
inputStream.skip(rangeStart);
}
byte[] buffer = new byte[65536];
long bytesRemaining = contentLength;
int bytesRead;
while (bytesRemaining > 0 && (bytesRead = inputStream.read(buffer, 0, (int) Math.min(buffer.length, bytesRemaining))) != -1) {
rawOut.write(buffer, 0, bytesRead);
bytesRemaining -= bytesRead;
}
}
// Stream finished
if (base64Out != null) {
base64Out.close(); // Also flushes and closes the wrapped gzipOut
} else if (gzipOut != null) {
gzipOut.close(); // Only close gzipOut if it wasn't wrapped by base64Out
} else {
rawOut.flush(); // Flush only the base output stream if nothing was wrapped
}
if (!response.isCommitted()) {
response.setStatus(HttpServletResponse.SC_OK);
response.getWriter().write(" ");
}
} catch (IOException e) {
// Streaming errors should not rethrow just log
LOGGER.warn(String.format("Streaming error for %s %s: %s", service, name, e.getMessage()));
} }
} catch (IOException | ApiException | DataException e) { // Encode the data if requested
LOGGER.warn(String.format("Unable to load %s %s: %s", service, name, e.getMessage())); if (encoding != null && Objects.equals(encoding.toLowerCase(), "base64")) {
if (!response.isCommitted()) { data = Base64.encode(data);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
}
} catch (NumberFormatException e) {
LOGGER.warn(String.format("Invalid range for %s %s: %s", service, name, e.getMessage()));
if (!response.isCommitted()) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage());
} }
response.addHeader("Accept-Ranges", "bytes");
response.setContentType(context.getMimeType(path.toString()));
response.setContentLength(data.length);
response.getOutputStream().write(data);
return response;
} catch (Exception e) {
LOGGER.debug(String.format("Unable to load %s %s: %s", service, name, e.getMessage()));
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
} }
} }
private FileProperties getFileProperties(Service service, String name, String identifier) { private FileProperties getFileProperties(Service service, String name, String identifier) {
try { try {
ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);

View File

@ -16,13 +16,9 @@ import org.qortal.api.model.AggregatedOrder;
import org.qortal.api.model.TradeWithOrderInfo; import org.qortal.api.model.TradeWithOrderInfo;
import org.qortal.api.resource.TransactionsResource.ConfirmationStatus; import org.qortal.api.resource.TransactionsResource.ConfirmationStatus;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.controller.hsqldb.HSQLDBBalanceRecorder;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
import org.qortal.data.account.AccountBalanceData; import org.qortal.data.account.AccountBalanceData;
import org.qortal.data.account.AccountData; import org.qortal.data.account.AccountData;
import org.qortal.data.account.AddressAmountData;
import org.qortal.data.account.BlockHeightRange;
import org.qortal.data.account.BlockHeightRangeAddressAmounts;
import org.qortal.data.asset.AssetData; import org.qortal.data.asset.AssetData;
import org.qortal.data.asset.OrderData; import org.qortal.data.asset.OrderData;
import org.qortal.data.asset.RecentTradeData; import org.qortal.data.asset.RecentTradeData;
@ -37,7 +33,6 @@ import org.qortal.transaction.Transaction;
import org.qortal.transaction.Transaction.ValidationResult; import org.qortal.transaction.Transaction.ValidationResult;
import org.qortal.transform.TransformationException; import org.qortal.transform.TransformationException;
import org.qortal.transform.transaction.*; import org.qortal.transform.transaction.*;
import org.qortal.utils.BalanceRecorderUtils;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
@ -47,7 +42,6 @@ import javax.ws.rs.core.MediaType;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Collections; import java.util.Collections;
import java.util.List; import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors; import java.util.stream.Collectors;
@Path("/assets") @Path("/assets")
@ -185,122 +179,6 @@ public class AssetsResource {
} }
} }
@GET
@Path("/balancedynamicranges")
@Operation(
summary = "Get balance dynamic ranges listed.",
description = ".",
responses = {
@ApiResponse(
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = BlockHeightRange.class
)
)
)
)
}
)
public List<BlockHeightRange> getBalanceDynamicRanges(
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
if( recorder.isPresent()) {
return recorder.get().getRanges(offset, limit, reverse);
}
else {
return new ArrayList<>(0);
}
}
@GET
@Path("/balancedynamicrange/{height}")
@Operation(
summary = "Get balance dynamic range for a given height.",
description = ".",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
implementation = BlockHeightRange.class
)
)
)
}
)
@ApiErrors({
ApiError.INVALID_CRITERIA, ApiError.INVALID_DATA
})
public BlockHeightRange getBalanceDynamicRange(@PathParam("height") int height) {
Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
if( recorder.isPresent()) {
Optional<BlockHeightRange> range = recorder.get().getRange(height);
if( range.isPresent() ) {
return range.get();
}
else {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
}
}
else {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
}
}
@GET
@Path("/balancedynamicamounts/{begin}/{end}")
@Operation(
summary = "Get balance dynamic ranges address amounts listed.",
description = ".",
responses = {
@ApiResponse(
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = AddressAmountData.class
)
)
)
)
}
)
@ApiErrors({
ApiError.INVALID_CRITERIA, ApiError.INVALID_DATA
})
public List<AddressAmountData> getBalanceDynamicAddressAmounts(
@PathParam("begin") int begin,
@PathParam("end") int end,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit) {
Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
if( recorder.isPresent()) {
Optional<BlockHeightRangeAddressAmounts> addressAmounts = recorder.get().getAddressAmounts(new BlockHeightRange(begin, end, false));
if( addressAmounts.isPresent() ) {
return addressAmounts.get().getAmounts().stream()
.sorted(BalanceRecorderUtils.ADDRESS_AMOUNT_DATA_COMPARATOR.reversed())
.skip(offset)
.limit(limit)
.collect(Collectors.toList());
}
else {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
}
}
else {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
}
}
@GET @GET
@Path("/openorders/{assetid}/{otherassetid}") @Path("/openorders/{assetid}/{otherassetid}")
@Operation( @Operation(

View File

@ -19,8 +19,6 @@ import org.qortal.crypto.Crypto;
import org.qortal.data.account.AccountData; import org.qortal.data.account.AccountData;
import org.qortal.data.block.BlockData; import org.qortal.data.block.BlockData;
import org.qortal.data.block.BlockSummaryData; import org.qortal.data.block.BlockSummaryData;
import org.qortal.data.block.DecodedOnlineAccountData;
import org.qortal.data.network.OnlineAccountData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
import org.qortal.repository.BlockArchiveReader; import org.qortal.repository.BlockArchiveReader;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
@ -29,7 +27,6 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.transform.TransformationException; import org.qortal.transform.TransformationException;
import org.qortal.transform.block.BlockTransformer; import org.qortal.transform.block.BlockTransformer;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import org.qortal.utils.Blocks;
import org.qortal.utils.Triple; import org.qortal.utils.Triple;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
@ -48,7 +45,6 @@ import java.util.ArrayList;
import java.util.Arrays; import java.util.Arrays;
import java.util.Comparator; import java.util.Comparator;
import java.util.List; import java.util.List;
import java.util.Set;
@Path("/blocks") @Path("/blocks")
@Tag(name = "Blocks") @Tag(name = "Blocks")
@ -546,7 +542,6 @@ public class BlocksResource {
} }
} }
String minterAddress = Account.getRewardShareMintingAddress(repository, blockData.getMinterPublicKey());
int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey()); int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey());
if (minterLevel == 0) if (minterLevel == 0)
// This may be unavailable when requesting a trimmed block // This may be unavailable when requesting a trimmed block
@ -559,7 +554,6 @@ public class BlocksResource {
BlockMintingInfo blockMintingInfo = new BlockMintingInfo(); BlockMintingInfo blockMintingInfo = new BlockMintingInfo();
blockMintingInfo.minterPublicKey = blockData.getMinterPublicKey(); blockMintingInfo.minterPublicKey = blockData.getMinterPublicKey();
blockMintingInfo.minterAddress = minterAddress;
blockMintingInfo.minterLevel = minterLevel; blockMintingInfo.minterLevel = minterLevel;
blockMintingInfo.onlineAccountsCount = blockData.getOnlineAccountsCount(); blockMintingInfo.onlineAccountsCount = blockData.getOnlineAccountsCount();
blockMintingInfo.maxDistance = new BigDecimal(block.MAX_DISTANCE); blockMintingInfo.maxDistance = new BigDecimal(block.MAX_DISTANCE);
@ -894,49 +888,4 @@ public class BlocksResource {
} }
} }
@GET
@Path("/onlineaccounts/{height}")
@Operation(
summary = "Get online accounts for block",
description = "Returns the online accounts who submitted signatures for this block",
responses = {
@ApiResponse(
description = "online accounts",
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = DecodedOnlineAccountData.class
)
)
)
)
}
)
@ApiErrors({
ApiError.BLOCK_UNKNOWN, ApiError.REPOSITORY_ISSUE
})
public Set<DecodedOnlineAccountData> getOnlineAccounts(@PathParam("height") int height) {
try (final Repository repository = RepositoryManager.getRepository()) {
// get block from database
BlockData blockData = repository.getBlockRepository().fromHeight(height);
// if block data is not in the database, then try the archive
if (blockData == null) {
blockData = repository.getBlockArchiveRepository().fromHeight(height);
// if the block is not in the database or the archive, then the block is unknown
if( blockData == null ) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
}
}
Set<DecodedOnlineAccountData> onlineAccounts = Blocks.getDecodedOnlineAccountsForBlock(repository, blockData);
return onlineAccounts;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE);
}
}
} }

View File

@ -234,16 +234,12 @@ public class ChatResource {
} }
) )
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE}) @ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
public ActiveChats getActiveChats( public ActiveChats getActiveChats(@PathParam("address") String address, @QueryParam("encoding") Encoding encoding) {
@PathParam("address") String address,
@QueryParam("encoding") Encoding encoding,
@QueryParam("haschatreference") Boolean hasChatReference
) {
if (address == null || !Crypto.isValidAddress(address)) if (address == null || !Crypto.isValidAddress(address))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
return repository.getChatRepository().getActiveChats(address, encoding, hasChatReference); return repository.getChatRepository().getActiveChats(address, encoding);
} catch (DataException e) { } catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
} }

View File

@ -502,10 +502,10 @@ public class CrossChainBitcoinResource {
} }
@GET @GET
@Path("/feerequired") @Path("/feeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking BTC to the trade offer creator.", summary = "Returns Bitcoin fee per Kb.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Returns Bitcoin fee per Kb.",
responses = { responses = {
@ApiResponse( @ApiResponse(
content = @Content( content = @Content(
@ -516,17 +516,17 @@ public class CrossChainBitcoinResource {
) )
} }
) )
public String getBitcoinFeeRequired() { public String getBitcoinFeeCeiling() {
Bitcoin bitcoin = Bitcoin.getInstance(); Bitcoin bitcoin = Bitcoin.getInstance();
return String.valueOf(bitcoin.getFeeRequired()); return String.valueOf(bitcoin.getFeeCeiling());
} }
@POST @POST
@Path("/updatefeerequired") @Path("/updatefeeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking BTC to the trade offer creator.", summary = "Sets Bitcoin fee ceiling.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Sets Bitcoin fee ceiling.",
requestBody = @RequestBody( requestBody = @RequestBody(
required = true, required = true,
content = @Content( content = @Content(
@ -545,13 +545,13 @@ public class CrossChainBitcoinResource {
} }
) )
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA}) @ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
public String setBitcoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) { public String setBitcoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
Bitcoin bitcoin = Bitcoin.getInstance(); Bitcoin bitcoin = Bitcoin.getInstance();
try { try {
return CrossChainUtils.setFeeRequired(bitcoin, fee); return CrossChainUtils.setFeeCeiling(bitcoin, fee);
} }
catch (IllegalArgumentException e) { catch (IllegalArgumentException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

View File

@ -502,10 +502,10 @@ public class CrossChainDigibyteResource {
} }
@GET @GET
@Path("/feerequired") @Path("/feeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking DGB to the trade offer creator.", summary = "Returns Digibyte fee per Kb.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Returns Digibyte fee per Kb.",
responses = { responses = {
@ApiResponse( @ApiResponse(
content = @Content( content = @Content(
@ -516,17 +516,17 @@ public class CrossChainDigibyteResource {
) )
} }
) )
public String getDigibyteFeeRequired() { public String getDigibyteFeeCeiling() {
Digibyte digibyte = Digibyte.getInstance(); Digibyte digibyte = Digibyte.getInstance();
return String.valueOf(digibyte.getFeeRequired()); return String.valueOf(digibyte.getFeeCeiling());
} }
@POST @POST
@Path("/updatefeerequired") @Path("/updatefeeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking DGB to the trade offer creator.", summary = "Sets Digibyte fee ceiling.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Sets Digibyte fee ceiling.",
requestBody = @RequestBody( requestBody = @RequestBody(
required = true, required = true,
content = @Content( content = @Content(
@ -545,13 +545,13 @@ public class CrossChainDigibyteResource {
} }
) )
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA}) @ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
public String setDigibyteFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) { public String setDigibyteFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
Digibyte digibyte = Digibyte.getInstance(); Digibyte digibyte = Digibyte.getInstance();
try { try {
return CrossChainUtils.setFeeRequired(digibyte, fee); return CrossChainUtils.setFeeCeiling(digibyte, fee);
} }
catch (IllegalArgumentException e) { catch (IllegalArgumentException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

View File

@ -502,10 +502,10 @@ public class CrossChainDogecoinResource {
} }
@GET @GET
@Path("/feerequired") @Path("/feeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking DOGE to the trade offer creator.", summary = "Returns Dogecoin fee per Kb.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Returns Dogecoin fee per Kb.",
responses = { responses = {
@ApiResponse( @ApiResponse(
content = @Content( content = @Content(
@ -516,17 +516,17 @@ public class CrossChainDogecoinResource {
) )
} }
) )
public String getDogecoinFeeRequired() { public String getDogecoinFeeCeiling() {
Dogecoin dogecoin = Dogecoin.getInstance(); Dogecoin dogecoin = Dogecoin.getInstance();
return String.valueOf(dogecoin.getFeeRequired()); return String.valueOf(dogecoin.getFeeCeiling());
} }
@POST @POST
@Path("/updatefeerequired") @Path("/updatefeeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking DOGE to the trade offer creator.", summary = "Sets Dogecoin fee ceiling.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Sets Dogecoin fee ceiling.",
requestBody = @RequestBody( requestBody = @RequestBody(
required = true, required = true,
content = @Content( content = @Content(
@ -545,13 +545,13 @@ public class CrossChainDogecoinResource {
} }
) )
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA}) @ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
public String setDogecoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) { public String setDogecoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
Dogecoin dogecoin = Dogecoin.getInstance(); Dogecoin dogecoin = Dogecoin.getInstance();
try { try {
return CrossChainUtils.setFeeRequired(dogecoin, fee); return CrossChainUtils.setFeeCeiling(dogecoin, fee);
} }
catch (IllegalArgumentException e) { catch (IllegalArgumentException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

View File

@ -157,7 +157,7 @@ public class CrossChainHtlcResource {
htlcStatus.bitcoinP2shAddress = p2shAddress; htlcStatus.bitcoinP2shAddress = p2shAddress;
htlcStatus.bitcoinP2shBalance = BigDecimal.valueOf(p2shBalance, 8); htlcStatus.bitcoinP2shBalance = BigDecimal.valueOf(p2shBalance, 8);
List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddress.toString(), false); List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddress.toString());
if (p2shBalance > 0L && !fundingOutputs.isEmpty()) { if (p2shBalance > 0L && !fundingOutputs.isEmpty()) {
htlcStatus.canRedeem = now >= medianBlockTime * 1000L; htlcStatus.canRedeem = now >= medianBlockTime * 1000L;
@ -401,7 +401,7 @@ public class CrossChainHtlcResource {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(decodedTradePrivateKey); ECKey redeemKey = ECKey.fromPrivate(decodedTradePrivateKey);
List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoiny.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoiny.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, decodedSecret, foreignBlockchainReceivingAccountInfo); fundingOutputs, redeemScriptA, decodedSecret, foreignBlockchainReceivingAccountInfo);
@ -664,7 +664,7 @@ public class CrossChainHtlcResource {
// ElectrumX coins // ElectrumX coins
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddressA);
// Validate the destination foreign blockchain address // Validate the destination foreign blockchain address
Address receiving = Address.fromString(bitcoiny.getNetworkParameters(), receiveAddress); Address receiving = Address.fromString(bitcoiny.getNetworkParameters(), receiveAddress);

View File

@ -540,10 +540,10 @@ public class CrossChainLitecoinResource {
} }
@GET @GET
@Path("/feerequired") @Path("/feeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking LTC to the trade offer creator.", summary = "Returns Litecoin fee per Kb.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Returns Litecoin fee per Kb.",
responses = { responses = {
@ApiResponse( @ApiResponse(
content = @Content( content = @Content(
@ -554,17 +554,17 @@ public class CrossChainLitecoinResource {
) )
} }
) )
public String getLitecoinFeeRequired() { public String getLitecoinFeeCeiling() {
Litecoin litecoin = Litecoin.getInstance(); Litecoin litecoin = Litecoin.getInstance();
return String.valueOf(litecoin.getFeeRequired()); return String.valueOf(litecoin.getFeeCeiling());
} }
@POST @POST
@Path("/updatefeerequired") @Path("/updatefeeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking LTC to the trade offer creator.", summary = "Sets Litecoin fee ceiling.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Sets Litecoin fee ceiling.",
requestBody = @RequestBody( requestBody = @RequestBody(
required = true, required = true,
content = @Content( content = @Content(
@ -583,13 +583,13 @@ public class CrossChainLitecoinResource {
} }
) )
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA}) @ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
public String setLitecoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) { public String setLitecoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
Litecoin litecoin = Litecoin.getInstance(); Litecoin litecoin = Litecoin.getInstance();
try { try {
return CrossChainUtils.setFeeRequired(litecoin, fee); return CrossChainUtils.setFeeCeiling(litecoin, fee);
} }
catch (IllegalArgumentException e) { catch (IllegalArgumentException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

View File

@ -587,10 +587,10 @@ public class CrossChainPirateChainResource {
} }
@GET @GET
@Path("/feerequired") @Path("/feeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking ARRR to the trade offer creator.", summary = "Returns PirateChain fee per Kb.",
description = "The total fee required for unlocking ARRR to the trade offer creator.", description = "Returns PirateChain fee per Kb.",
responses = { responses = {
@ApiResponse( @ApiResponse(
content = @Content( content = @Content(
@ -601,17 +601,17 @@ public class CrossChainPirateChainResource {
) )
} }
) )
public String getPirateChainFeeRequired() { public String getPirateChainFeeCeiling() {
PirateChain pirateChain = PirateChain.getInstance(); PirateChain pirateChain = PirateChain.getInstance();
return String.valueOf(pirateChain.getFeeRequired()); return String.valueOf(pirateChain.getFeeCeiling());
} }
@POST @POST
@Path("/updatefeerequired") @Path("/updatefeeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking ARRR to the trade offer creator.", summary = "Sets PirateChain fee ceiling.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Sets PirateChain fee ceiling.",
requestBody = @RequestBody( requestBody = @RequestBody(
required = true, required = true,
content = @Content( content = @Content(
@ -630,13 +630,13 @@ public class CrossChainPirateChainResource {
} }
) )
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA}) @ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
public String setPirateChainFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) { public String setPirateChainFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
PirateChain pirateChain = PirateChain.getInstance(); PirateChain pirateChain = PirateChain.getInstance();
try { try {
return CrossChainUtils.setFeeRequired(pirateChain, fee); return CrossChainUtils.setFeeCeiling(pirateChain, fee);
} }
catch (IllegalArgumentException e) { catch (IllegalArgumentException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

View File

@ -502,10 +502,10 @@ public class CrossChainRavencoinResource {
} }
@GET @GET
@Path("/feerequired") @Path("/feeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking RVN to the trade offer creator.", summary = "Returns Ravencoin fee per Kb.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Returns Ravencoin fee per Kb.",
responses = { responses = {
@ApiResponse( @ApiResponse(
content = @Content( content = @Content(
@ -516,17 +516,17 @@ public class CrossChainRavencoinResource {
) )
} }
) )
public String getRavencoinFeeRequired() { public String getRavencoinFeeCeiling() {
Ravencoin ravencoin = Ravencoin.getInstance(); Ravencoin ravencoin = Ravencoin.getInstance();
return String.valueOf(ravencoin.getFeeRequired()); return String.valueOf(ravencoin.getFeeCeiling());
} }
@POST @POST
@Path("/updatefeerequired") @Path("/updatefeeceiling")
@Operation( @Operation(
summary = "The total fee required for unlocking RVN to the trade offer creator.", summary = "Sets Ravencoin fee ceiling.",
description = "This is in sats for a transaction that is approximately 300 kB in size.", description = "Sets Ravencoin fee ceiling.",
requestBody = @RequestBody( requestBody = @RequestBody(
required = true, required = true,
content = @Content( content = @Content(
@ -545,13 +545,13 @@ public class CrossChainRavencoinResource {
} }
) )
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA}) @ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
public String setRavencoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) { public String setRavencoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
Ravencoin ravencoin = Ravencoin.getInstance(); Ravencoin ravencoin = Ravencoin.getInstance();
try { try {
return CrossChainUtils.setFeeRequired(ravencoin, fee); return CrossChainUtils.setFeeCeiling(ravencoin, fee);
} }
catch (IllegalArgumentException e) { catch (IllegalArgumentException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

View File

@ -10,17 +10,12 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody;
import io.swagger.v3.oas.annotations.responses.ApiResponse; import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.security.SecurityRequirement; import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag; import io.swagger.v3.oas.annotations.tags.Tag;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.glassfish.jersey.media.multipart.ContentDisposition;
import org.qortal.api.ApiError; import org.qortal.api.ApiError;
import org.qortal.api.ApiErrors; import org.qortal.api.ApiErrors;
import org.qortal.api.ApiExceptionFactory; import org.qortal.api.ApiExceptionFactory;
import org.qortal.api.Security; import org.qortal.api.Security;
import org.qortal.api.model.CrossChainCancelRequest; import org.qortal.api.model.CrossChainCancelRequest;
import org.qortal.api.model.CrossChainTradeLedgerEntry;
import org.qortal.api.model.CrossChainTradeSummary; import org.qortal.api.model.CrossChainTradeSummary;
import org.qortal.controller.ForeignFeesManager;
import org.qortal.controller.tradebot.TradeBot; import org.qortal.controller.tradebot.TradeBot;
import org.qortal.crosschain.ACCT; import org.qortal.crosschain.ACCT;
import org.qortal.crosschain.AcctMode; import org.qortal.crosschain.AcctMode;
@ -32,8 +27,6 @@ import org.qortal.data.at.ATData;
import org.qortal.data.at.ATStateData; import org.qortal.data.at.ATStateData;
import org.qortal.data.crosschain.CrossChainTradeData; import org.qortal.data.crosschain.CrossChainTradeData;
import org.qortal.data.crosschain.TransactionSummary; import org.qortal.data.crosschain.TransactionSummary;
import org.qortal.data.crosschain.ForeignFeeDecodedData;
import org.qortal.data.crosschain.ForeignFeeEncodedData;
import org.qortal.data.transaction.BaseTransactionData; import org.qortal.data.transaction.BaseTransactionData;
import org.qortal.data.transaction.MessageTransactionData; import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
@ -51,36 +44,21 @@ import org.qortal.utils.Base58;
import org.qortal.utils.ByteArray; import org.qortal.utils.ByteArray;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.*; import javax.ws.rs.*;
import javax.ws.rs.core.Context; import javax.ws.rs.core.Context;
import javax.ws.rs.core.HttpHeaders;
import javax.ws.rs.core.MediaType; import javax.ws.rs.core.MediaType;
import java.io.IOException;
import java.util.*; import java.util.*;
import java.util.function.Supplier; import java.util.function.Supplier;
import java.util.stream.Collectors; import java.util.stream.Collectors;
@Path("/crosschain") @Path("/crosschain")
@Tag(name = "Cross-Chain") @Tag(name = "Cross-Chain")
public class CrossChainResource { public class CrossChainResource {
private static final Logger LOGGER = LogManager.getLogger(CrossChainResource.class);
@Context @Context
HttpServletRequest request; HttpServletRequest request;
@Context
HttpServletResponse response;
@Context
ServletContext context;
@GET @GET
@Path("/tradeoffers") @Path("/tradeoffers")
@Operation( @Operation(
@ -277,12 +255,6 @@ public class CrossChainResource {
description = "Only return trades that completed on/after this timestamp (milliseconds since epoch)", description = "Only return trades that completed on/after this timestamp (milliseconds since epoch)",
example = "1597310000000" example = "1597310000000"
) @QueryParam("minimumTimestamp") Long minimumTimestamp, ) @QueryParam("minimumTimestamp") Long minimumTimestamp,
@Parameter(
description = "Optionally filter by buyer Qortal public key"
) @QueryParam("buyerPublicKey") String buyerPublicKey58,
@Parameter(
description = "Optionally filter by seller Qortal public key"
) @QueryParam("sellerPublicKey") String sellerPublicKey58,
@Parameter( ref = "limit") @QueryParam("limit") Integer limit, @Parameter( ref = "limit") @QueryParam("limit") Integer limit,
@Parameter( ref = "offset" ) @QueryParam("offset") Integer offset, @Parameter( ref = "offset" ) @QueryParam("offset") Integer offset,
@Parameter( ref = "reverse" ) @QueryParam("reverse") Boolean reverse) { @Parameter( ref = "reverse" ) @QueryParam("reverse") Boolean reverse) {
@ -294,10 +266,6 @@ public class CrossChainResource {
if (minimumTimestamp != null && minimumTimestamp <= 0) if (minimumTimestamp != null && minimumTimestamp <= 0)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
// Decode public keys
byte[] buyerPublicKey = decodePublicKey(buyerPublicKey58);
byte[] sellerPublicKey = decodePublicKey(sellerPublicKey58);
final Boolean isFinished = Boolean.TRUE; final Boolean isFinished = Boolean.TRUE;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
@ -328,7 +296,7 @@ public class CrossChainResource {
byte[] codeHash = acctInfo.getKey().value; byte[] codeHash = acctInfo.getKey().value;
ACCT acct = acctInfo.getValue().get(); ACCT acct = acctInfo.getValue().get();
List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash, buyerPublicKey, sellerPublicKey, List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
isFinished, acct.getModeByteOffset(), (long) AcctMode.REDEEMED.value, minimumFinalHeight, isFinished, acct.getModeByteOffset(), (long) AcctMode.REDEEMED.value, minimumFinalHeight,
limit, offset, reverse); limit, offset, reverse);
@ -367,215 +335,6 @@ public class CrossChainResource {
} }
} }
@POST
@Path("/signedfees")
@Operation(
summary = "",
description = "",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
array = @ArraySchema(
schema = @Schema(
implementation = ForeignFeeEncodedData.class
)
)
)
),
responses = {
@ApiResponse(
description = "true on success",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "boolean"
)
)
)
}
)
public String postSignedForeignFees(List<ForeignFeeEncodedData> signedFees) {
LOGGER.info("signedFees = " + signedFees);
try {
ForeignFeesManager.getInstance().addSignedFees(signedFees);
return "true";
}
catch( Exception e ) {
LOGGER.error(e.getMessage(), e);
return "false";
}
}
@GET
@Path("/unsignedfees/{address}")
@Operation(
summary = "",
description = "",
responses = {
@ApiResponse(
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = ForeignFeeEncodedData.class
)
)
)
)
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
public List<ForeignFeeEncodedData> getUnsignedFees(@PathParam("address") String address) {
List<ForeignFeeEncodedData> unsignedFeesForAddress = ForeignFeesManager.getInstance().getUnsignedFeesForAddress(address);
LOGGER.info("address = " + address);
LOGGER.info("returning unsigned = " + unsignedFeesForAddress);
return unsignedFeesForAddress;
}
@GET
@Path("/signedfees")
@Operation(
summary = "",
description = "",
responses = {
@ApiResponse(
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = ForeignFeeDecodedData.class
)
)
)
)
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
public List<ForeignFeeDecodedData> getSignedFees() {
return ForeignFeesManager.getInstance().getSignedFees();
}
/**
* Decode Public Key
*
* @param publicKey58 the public key in a string
*
* @return the public key in bytes
*/
private byte[] decodePublicKey(String publicKey58) {
if( publicKey58 == null ) return null;
if( publicKey58.isEmpty() ) return new byte[0];
byte[] publicKey;
try {
publicKey = Base58.decode(publicKey58);
} catch (NumberFormatException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PUBLIC_KEY, e);
}
// Correct size for public key?
if (publicKey.length != Transformer.PUBLIC_KEY_LENGTH)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PUBLIC_KEY);
return publicKey;
}
@GET
@Path("/ledger/{publicKey}")
@Operation(
summary = "Accounting entries for all trades.",
description = "Returns accounting entries for all completed cross-chain trades",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "string",
format = "byte"
)
)
)
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
public HttpServletResponse getLedgerEntries(
@PathParam("publicKey") String publicKey58,
@Parameter(
description = "Only return trades that completed on/after this timestamp (milliseconds since epoch)",
example = "1597310000000"
) @QueryParam("minimumTimestamp") Long minimumTimestamp) {
byte[] publicKey = decodePublicKey(publicKey58);
// minimumTimestamp (if given) needs to be positive
if (minimumTimestamp != null && minimumTimestamp <= 0)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
try (final Repository repository = RepositoryManager.getRepository()) {
Integer minimumFinalHeight = null;
if (minimumTimestamp != null) {
minimumFinalHeight = repository.getBlockRepository().getHeightFromTimestamp(minimumTimestamp);
// If not found in the block repository it will return either 0 or 1
if (minimumFinalHeight == 0 || minimumFinalHeight == 1) {
// Try the archive
minimumFinalHeight = repository.getBlockArchiveRepository().getHeightFromTimestamp(minimumTimestamp);
}
if (minimumFinalHeight == 0)
// We don't have any blocks since minimumTimestamp, let alone trades, so nothing to return
return response;
// height returned from repository is for block BEFORE timestamp
// but we want trades AFTER timestamp so bump height accordingly
minimumFinalHeight++;
}
List<CrossChainTradeLedgerEntry> crossChainTradeLedgerEntries = new ArrayList<>();
Map<ByteArray, Supplier<ACCT>> acctsByCodeHash = SupportedBlockchain.getAcctMap();
// collect ledger entries for each ACCT
for (Map.Entry<ByteArray, Supplier<ACCT>> acctInfo : acctsByCodeHash.entrySet()) {
byte[] codeHash = acctInfo.getKey().value;
ACCT acct = acctInfo.getValue().get();
// collect buys and sells
CrossChainUtils.collectLedgerEntries(publicKey, repository, minimumFinalHeight, crossChainTradeLedgerEntries, codeHash, acct, true);
CrossChainUtils.collectLedgerEntries(publicKey, repository, minimumFinalHeight, crossChainTradeLedgerEntries, codeHash, acct, false);
}
crossChainTradeLedgerEntries.sort((a, b) -> Longs.compare(a.getTradeTimestamp(), b.getTradeTimestamp()));
response.setStatus(HttpServletResponse.SC_OK);
response.setContentType("text/csv");
response.setHeader(
HttpHeaders.CONTENT_DISPOSITION,
ContentDisposition
.type("attachment")
.fileName(CrossChainUtils.createLedgerFileName(Crypto.toAddress(publicKey)))
.build()
.toString()
);
CrossChainUtils.writeToLedger( response.getWriter(), crossChainTradeLedgerEntries);
return response;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
} catch (IOException e) {
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
return response;
}
}
@GET @GET
@Path("/price/{blockchain}") @Path("/price/{blockchain}")
@Operation( @Operation(

View File

@ -17,16 +17,13 @@ import org.qortal.api.ApiExceptionFactory;
import org.qortal.api.Security; import org.qortal.api.Security;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.model.crosschain.TradeBotRespondRequest; import org.qortal.api.model.crosschain.TradeBotRespondRequest;
import org.qortal.api.model.crosschain.TradeBotRespondRequests;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.controller.Controller; import org.qortal.controller.Controller;
import org.qortal.controller.tradebot.AcctTradeBot; import org.qortal.controller.tradebot.AcctTradeBot;
import org.qortal.controller.tradebot.TradeBot; import org.qortal.controller.tradebot.TradeBot;
import org.qortal.crosschain.ACCT; import org.qortal.crosschain.ACCT;
import org.qortal.crosschain.AcctMode; import org.qortal.crosschain.AcctMode;
import org.qortal.crosschain.Bitcoiny;
import org.qortal.crosschain.ForeignBlockchain; import org.qortal.crosschain.ForeignBlockchain;
import org.qortal.crosschain.PirateChain;
import org.qortal.crosschain.SupportedBlockchain; import org.qortal.crosschain.SupportedBlockchain;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
import org.qortal.data.at.ATData; import org.qortal.data.at.ATData;
@ -45,10 +42,8 @@ import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.*; import javax.ws.rs.*;
import javax.ws.rs.core.Context; import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType; import javax.ws.rs.core.MediaType;
import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.Objects; import java.util.Objects;
import java.util.Optional;
import java.util.stream.Collectors; import java.util.stream.Collectors;
@Path("/crosschain/tradebot") @Path("/crosschain/tradebot")
@ -192,39 +187,6 @@ public class CrossChainTradeBotResource {
public String tradeBotResponder(@HeaderParam(Security.API_KEY_HEADER) String apiKey, TradeBotRespondRequest tradeBotRespondRequest) { public String tradeBotResponder(@HeaderParam(Security.API_KEY_HEADER) String apiKey, TradeBotRespondRequest tradeBotRespondRequest) {
Security.checkApiCallAllowed(request); Security.checkApiCallAllowed(request);
return createTradeBotResponse(tradeBotRespondRequest);
}
@POST
@Path("/respondmultiple")
@Operation(
summary = "Respond to multiple trade offers. NOTE: WILL SPEND FUNDS!)",
description = "Start a new trade-bot entry to respond to chosen trade offers. Pirate Chain is not supported and will throw an invalid criteria error.",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(
implementation = TradeBotRespondRequests.class
)
)
),
responses = {
@ApiResponse(
content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
)
}
)
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_ADDRESS, ApiError.INVALID_CRITERIA, ApiError.FOREIGN_BLOCKCHAIN_BALANCE_ISSUE, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE, ApiError.REPOSITORY_ISSUE})
@SuppressWarnings("deprecation")
@SecurityRequirement(name = "apiKey")
public String tradeBotResponderMultiple(@HeaderParam(Security.API_KEY_HEADER) String apiKey, TradeBotRespondRequests tradeBotRespondRequest) {
Security.checkApiCallAllowed(request);
return createTradeBotResponseMultiple(tradeBotRespondRequest);
}
private String createTradeBotResponse(TradeBotRespondRequest tradeBotRespondRequest) {
final String atAddress = tradeBotRespondRequest.atAddress; final String atAddress = tradeBotRespondRequest.atAddress;
// We prefer foreignKey to deprecated xprv58 // We prefer foreignKey to deprecated xprv58
@ -295,99 +257,6 @@ public class CrossChainTradeBotResource {
} }
} }
private String createTradeBotResponseMultiple(TradeBotRespondRequests respondRequests) {
try (final Repository repository = RepositoryManager.getRepository()) {
if (respondRequests.foreignKey == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
List<CrossChainTradeData> crossChainTradeDataList = new ArrayList<>(respondRequests.addresses.size());
Optional<ACCT> acct = Optional.empty();
for(String atAddress : respondRequests.addresses ) {
if (atAddress == null || !Crypto.isValidAtAddress(atAddress))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
if (respondRequests.receivingAddress == null || !Crypto.isValidAddress(respondRequests.receivingAddress))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
if (!Controller.getInstance().isUpToDate(minLatestBlockTimestamp))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCKCHAIN_NEEDS_SYNC);
// Extract data from cross-chain trading AT
ATData atData = fetchAtDataWithChecking(repository, atAddress);
// TradeBot uses AT's code hash to map to ACCT
ACCT acctUsingAtData = TradeBot.getInstance().getAcctUsingAtData(atData);
if (acctUsingAtData == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
// if the optional is empty,
// then ensure the ACCT blockchain is a Bitcoiny blockchain, but not Pirate Chain and fill the optional
// Even though the Pirate Chain protocol does support multi send,
// the Pirate Chain API we are using does not support multi send
else if( acct.isEmpty() ) {
if( !(acctUsingAtData.getBlockchain() instanceof Bitcoiny) ||
acctUsingAtData.getBlockchain() instanceof PirateChain )
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
acct = Optional.of(acctUsingAtData);
}
// if the optional is filled, then ensure it is equal to the AT in this iteration
else if( !acctUsingAtData.getCodeBytesHash().equals(acct.get().getCodeBytesHash()) )
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
if (!acctUsingAtData.getBlockchain().isValidWalletKey(respondRequests.foreignKey))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
CrossChainTradeData crossChainTradeData = acctUsingAtData.populateTradeData(repository, atData);
if (crossChainTradeData == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
if (crossChainTradeData.mode != AcctMode.OFFERING)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
// Check if there is a buy or a cancel request in progress for this trade
List<Transaction.TransactionType> txTypes = List.of(Transaction.TransactionType.MESSAGE);
List<TransactionData> unconfirmed = repository.getTransactionRepository().getUnconfirmedTransactions(txTypes, null, 0, 0, false);
for (TransactionData transactionData : unconfirmed) {
MessageTransactionData messageTransactionData = (MessageTransactionData) transactionData;
if (Objects.equals(messageTransactionData.getRecipient(), atAddress)) {
// There is a pending request for this trade, so block this buy attempt to reduce the risk of refunds
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Trade has an existing buy request or is pending cancellation.");
}
}
crossChainTradeDataList.add(crossChainTradeData);
}
AcctTradeBot.ResponseResult result
= TradeBot.getInstance().startResponseMultiple(
repository,
acct.get(),
crossChainTradeDataList,
respondRequests.receivingAddress,
respondRequests.foreignKey,
(Bitcoiny) acct.get().getBlockchain());
switch (result) {
case OK:
return "true";
case BALANCE_ISSUE:
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_BALANCE_ISSUE);
case NETWORK_ISSUE:
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
default:
return "false";
}
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage());
}
}
@DELETE @DELETE
@Operation( @Operation(
summary = "Delete completed trade", summary = "Delete completed trade",

View File

@ -1,6 +1,5 @@
package org.qortal.api.resource; package org.qortal.api.resource;
import com.google.common.primitives.Bytes;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.bitcoinj.core.Address; import org.bitcoinj.core.Address;
@ -8,39 +7,19 @@ import org.bitcoinj.core.Coin;
import org.bitcoinj.script.Script; import org.bitcoinj.script.Script;
import org.bitcoinj.script.ScriptBuilder; import org.bitcoinj.script.ScriptBuilder;
import org.bouncycastle.util.Strings;
import org.json.simple.JSONObject;
import org.qortal.api.model.CrossChainTradeLedgerEntry;
import org.qortal.api.model.crosschain.BitcoinyTBDRequest;
import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.data.account.AccountBalanceData;
import org.qortal.data.at.ATData; import org.qortal.data.at.ATData;
import org.qortal.data.at.ATStateData;
import org.qortal.data.crosschain.*; import org.qortal.data.crosschain.*;
import org.qortal.event.EventBus;
import org.qortal.event.LockingFeeUpdateEvent;
import org.qortal.event.RequiredFeeUpdateEvent;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
import org.qortal.repository.Repository; import org.qortal.repository.Repository;
import org.qortal.utils.Amounts;
import org.qortal.utils.BitTwiddling;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.Writer;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*; import java.util.*;
import java.util.function.Function;
import java.util.stream.Collectors; import java.util.stream.Collectors;
public class CrossChainUtils { public class CrossChainUtils {
public static final String QORT_CURRENCY_CODE = "QORT";
private static final Logger LOGGER = LogManager.getLogger(CrossChainUtils.class); private static final Logger LOGGER = LogManager.getLogger(CrossChainUtils.class);
public static final String CORE_API_CALL = "Core API Call"; public static final String CORE_API_CALL = "Core API Call";
public static final String QORTAL_EXCHANGE_LABEL = "Qortal";
public static ServerConfigurationInfo buildServerConfigurationInfo(Bitcoiny blockchain) { public static ServerConfigurationInfo buildServerConfigurationInfo(Bitcoiny blockchain) {
@ -104,13 +83,11 @@ public class CrossChainUtils {
bitcoiny.setFeePerKb(Coin.valueOf(satoshis) ); bitcoiny.setFeePerKb(Coin.valueOf(satoshis) );
EventBus.INSTANCE.notify(new LockingFeeUpdateEvent());
return String.valueOf(bitcoiny.getFeePerKb().value); return String.valueOf(bitcoiny.getFeePerKb().value);
} }
/** /**
* Set Fee Required * Set Fee Ceiling
* *
* @param bitcoiny the blockchain support * @param bitcoiny the blockchain support
* @param fee the fee in satoshis * @param fee the fee in satoshis
@ -119,16 +96,14 @@ public class CrossChainUtils {
* *
* @throws IllegalArgumentException if invalid * @throws IllegalArgumentException if invalid
*/ */
public static String setFeeRequired(Bitcoiny bitcoiny, String fee) throws IllegalArgumentException{ public static String setFeeCeiling(Bitcoiny bitcoiny, String fee) throws IllegalArgumentException{
long satoshis = Long.parseLong(fee); long satoshis = Long.parseLong(fee);
if( satoshis < 0 ) throw new IllegalArgumentException("can't set fee to negative number"); if( satoshis < 0 ) throw new IllegalArgumentException("can't set fee to negative number");
bitcoiny.setFeeRequired( Long.parseLong(fee)); bitcoiny.setFeeCeiling( Long.parseLong(fee));
EventBus.INSTANCE.notify(new RequiredFeeUpdateEvent(bitcoiny)); return String.valueOf(bitcoiny.getFeeCeiling());
return String.valueOf(bitcoiny.getFeeRequired());
} }
/** /**
@ -237,9 +212,6 @@ public class CrossChainUtils {
return bitcoiny.getBlockchainProvider().removeServer(server); return bitcoiny.getBlockchainProvider().removeServer(server);
} }
public static ChainableServer getCurrentServer( Bitcoiny bitcoiny ) {
return bitcoiny.getBlockchainProvider().getCurrentServer();
}
/** /**
* Set Current Server * Set Current Server
* *
@ -573,252 +545,4 @@ public class CrossChainUtils {
server.getConnectionType().toString(), server.getConnectionType().toString(),
false); false);
} }
/**
* Get Bitcoiny TBD (To Be Determined)
*
* @param bitcoinyTBDRequest the parameters for the Bitcoiny TBD
* @return the Bitcoiny TBD
* @throws DataException
*/
public static BitcoinyTBD getBitcoinyTBD(BitcoinyTBDRequest bitcoinyTBDRequest) throws DataException {
try {
DeterminedNetworkParams networkParams = new DeterminedNetworkParams(bitcoinyTBDRequest);
BitcoinyTBD bitcoinyTBD
= BitcoinyTBD.getInstance(bitcoinyTBDRequest.getCode())
.orElse(BitcoinyTBD.buildInstance(
bitcoinyTBDRequest,
networkParams)
);
return bitcoinyTBD;
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
return null;
}
/**
* Get Version Decimal
*
* @param jsonObject the JSON object with the version attribute
* @param attribute the attribute that hold the version value
* @return the version as a decimal number, discarding
* @throws NumberFormatException
*/
public static double getVersionDecimal(JSONObject jsonObject, String attribute) throws NumberFormatException {
String versionString = (String) jsonObject.get(attribute);
return Double.parseDouble(reduceDelimeters(versionString, 1, '.'));
}
/**
* Reduce Delimeters
*
* @param value the raw string
* @param max the max number of the delimeter
* @param delimeter the delimeter
* @return the processed value with the max number of delimeters
*/
public static String reduceDelimeters(String value, int max, char delimeter) {
if( max < 1 ) return value;
String[] splits = Strings.split(value, delimeter);
StringBuffer buffer = new StringBuffer(splits[0]);
int limit = Math.min(max + 1, splits.length);
for( int index = 1; index < limit; index++) {
buffer.append(delimeter);
buffer.append(splits[index]);
}
return buffer.toString();
}
/** Returns
/**
* Build Offer Message
*
* @param partnerBitcoinPKH
* @param hashOfSecretA
* @param lockTimeA
* @return 'offer' MESSAGE payload for trade partner to send to AT creator's trade address
*/
public static byte[] buildOfferMessage(byte[] partnerBitcoinPKH, byte[] hashOfSecretA, int lockTimeA) {
byte[] lockTimeABytes = BitTwiddling.toBEByteArray((long) lockTimeA);
return Bytes.concat(partnerBitcoinPKH, hashOfSecretA, lockTimeABytes);
}
/**
* Write To Ledger
*
* @param writer the writer to the ledger
* @param entries the entries to write to the ledger
*
* @throws IOException
*/
public static void writeToLedger(Writer writer, List<CrossChainTradeLedgerEntry> entries) throws IOException {
BufferedWriter bufferedWriter = new BufferedWriter(writer);
StringJoiner header = new StringJoiner(",");
header.add("Market");
header.add("Currency");
header.add("Quantity");
header.add("Commission Paid");
header.add("Commission Currency");
header.add("Total Price");
header.add("Date Time");
header.add("Exchange");
bufferedWriter.append(header.toString());
DateFormat dateFormatter = new SimpleDateFormat("yyyyMMdd HH:mm");
dateFormatter.setTimeZone(TimeZone.getTimeZone("UTC"));
for( CrossChainTradeLedgerEntry entry : entries ) {
StringJoiner joiner = new StringJoiner(",");
joiner.add(entry.getMarket());
joiner.add(entry.getCurrency());
joiner.add(String.valueOf(Amounts.prettyAmount(entry.getQuantity())));
joiner.add(String.valueOf(Amounts.prettyAmount(entry.getFeeAmount())));
joiner.add(entry.getFeeCurrency());
joiner.add(String.valueOf(Amounts.prettyAmount(entry.getTotalPrice())));
joiner.add(dateFormatter.format(new Date(entry.getTradeTimestamp())));
joiner.add(QORTAL_EXCHANGE_LABEL);
bufferedWriter.newLine();
bufferedWriter.append(joiner.toString());
}
bufferedWriter.newLine();
bufferedWriter.flush();
}
/**
* Create Ledger File Name
*
* Create a file name the includes timestamp and address.
*
* @param address the address
*
* @return the file name created
*/
public static String createLedgerFileName(String address) {
DateFormat dateFormatter = new SimpleDateFormat("yyyyMMddHHmmss");
String fileName = "ledger-" + address + "-" + dateFormatter.format(new Date());
return fileName;
}
/**
* Collect Ledger Entries
*
* @param publicKey the public key for the ledger entries, buy and sell
* @param repository the data repository
* @param minimumFinalHeight the minimum block height for entries to be collected
* @param entries the ledger entries to add to
* @param codeHash code hash for the entry blockchain
* @param acct the ACCT for the entry blockchain
* @param isBuy true collecting entries for a buy, otherwise false
*
* @throws DataException
*/
public static void collectLedgerEntries(
byte[] publicKey,
Repository repository,
Integer minimumFinalHeight,
List<CrossChainTradeLedgerEntry> entries,
byte[] codeHash,
ACCT acct,
boolean isBuy) throws DataException {
// get all the final AT states for the code hash (foreign coin)
List<ATStateData> atStates
= repository.getATRepository().getMatchingFinalATStates(
codeHash,
isBuy ? publicKey : null,
!isBuy ? publicKey : null,
Boolean.TRUE, acct.getModeByteOffset(),
(long) AcctMode.REDEEMED.value,
minimumFinalHeight,
null, null, false
);
String foreignBlockchainCurrencyCode = acct.getBlockchain().getCurrencyCode();
// for each trade, build ledger entry, collect ledger entry
for (ATStateData atState : atStates) {
CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState);
// We also need block timestamp for use as trade timestamp
long localTimestamp = repository.getBlockRepository().getTimestampFromHeight(atState.getHeight());
if (localTimestamp == 0) {
// Try the archive
localTimestamp = repository.getBlockArchiveRepository().getTimestampFromHeight(atState.getHeight());
}
CrossChainTradeLedgerEntry ledgerEntry
= new CrossChainTradeLedgerEntry(
isBuy ? QORT_CURRENCY_CODE : foreignBlockchainCurrencyCode,
isBuy ? foreignBlockchainCurrencyCode : QORT_CURRENCY_CODE,
isBuy ? crossChainTradeData.qortAmount : crossChainTradeData.expectedForeignAmount,
0,
foreignBlockchainCurrencyCode,
isBuy ? crossChainTradeData.expectedForeignAmount : crossChainTradeData.qortAmount,
localTimestamp);
entries.add(ledgerEntry);
}
}
public static List<CrossChainTradeData> populateTradeDataList(Repository repository, ACCT acct, List<ATData> atDataList) throws DataException {
if(atDataList.isEmpty()) return new ArrayList<>(0);
List<ATStateData> latestATStates
= repository.getATRepository()
.getLatestATStates(
atDataList.stream()
.map(ATData::getATAddress)
.collect(Collectors.toList())
);
Map<String, ATStateData> atStateDataByAtAddress
= latestATStates.stream().collect(Collectors.toMap(ATStateData::getATAddress, Function.identity()));
Map<String, ATData> atDataByAtAddress
= atDataList.stream().collect(Collectors.toMap(ATData::getATAddress, Function.identity()));
Map<String, Long> balanceByAtAddress
= repository
.getAccountRepository()
.getBalances(new ArrayList<>(atDataByAtAddress.keySet()), Asset.QORT)
.stream().collect(Collectors.toMap(AccountBalanceData::getAddress, AccountBalanceData::getBalance));
List<CrossChainTradeData> crossChainTradeDataList = new ArrayList<>(latestATStates.size());
for( ATStateData atStateData : latestATStates ) {
ATData atData = atDataByAtAddress.get(atStateData.getATAddress());
crossChainTradeDataList.add(
acct.populateTradeData(
repository,
atData.getCreatorPublicKey(),
atData.getCreation(),
atStateData,
OptionalLong.of(balanceByAtAddress.get(atStateData.getATAddress()))
)
);
}
return crossChainTradeDataList;
}
} }

View File

@ -33,7 +33,6 @@ import javax.ws.rs.*;
import javax.ws.rs.core.Context; import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType; import javax.ws.rs.core.MediaType;
import java.util.List; import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors; import java.util.stream.Collectors;
@Path("/names") @Path("/names")
@ -105,45 +104,6 @@ public class NamesResource {
} }
} }
@GET
@Path("/primary/{address}")
@Operation(
summary = "primary name owned by address",
responses = {
@ApiResponse(
description = "registered primary name info",
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(implementation = NameSummary.class)
)
)
}
)
@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE, ApiError.UNAUTHORIZED})
public NameSummary getPrimaryNameByAddress(@PathParam("address") String address) {
if (!Crypto.isValidAddress(address))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
try (final Repository repository = RepositoryManager.getRepository()) {
if (Settings.getInstance().isLite()) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.UNAUTHORIZED);
}
else {
Optional<String> primaryName = repository.getNameRepository().getPrimaryName(address);
if(primaryName.isPresent()) {
return new NameSummary(new NameData(primaryName.get(), address));
}
else {
return new NameSummary((new NameData(null, address)));
}
}
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET @GET
@Path("/{name}") @Path("/{name}")
@Operation( @Operation(

View File

@ -32,16 +32,13 @@ import org.qortal.controller.Synchronizer.SynchronizationResult;
import org.qortal.controller.repository.BlockArchiveRebuilder; import org.qortal.controller.repository.BlockArchiveRebuilder;
import org.qortal.data.account.MintingAccountData; import org.qortal.data.account.MintingAccountData;
import org.qortal.data.account.RewardShareData; import org.qortal.data.account.RewardShareData;
import org.qortal.data.system.DbConnectionInfo;
import org.qortal.network.Network; import org.qortal.network.Network;
import org.qortal.network.Peer; import org.qortal.network.Peer;
import org.qortal.network.PeerAddress; import org.qortal.network.PeerAddress;
import org.qortal.repository.ReindexManager;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
import org.qortal.repository.Repository; import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager; import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.data.system.SystemInfo;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
@ -54,7 +51,6 @@ import java.net.InetSocketAddress;
import java.net.UnknownHostException; import java.net.UnknownHostException;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException; import java.util.concurrent.TimeoutException;
@ -462,7 +458,7 @@ public class AdminResource {
// Qortal: check reward-share's minting account is still allowed to mint // Qortal: check reward-share's minting account is still allowed to mint
Account rewardShareMintingAccount = new Account(repository, rewardShareData.getMinter()); Account rewardShareMintingAccount = new Account(repository, rewardShareData.getMinter());
if (!rewardShareMintingAccount.canMint(false)) if (!rewardShareMintingAccount.canMint())
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.CANNOT_MINT); throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.CANNOT_MINT);
MintingAccountData mintingAccountData = new MintingAccountData(mintingAccount.getPrivateKey(), mintingAccount.getPublicKey()); MintingAccountData mintingAccountData = new MintingAccountData(mintingAccount.getPrivateKey(), mintingAccount.getPublicKey());
@ -898,50 +894,6 @@ public class AdminResource {
} }
} }
@POST
@Path("/repository/reindex")
@Operation(
summary = "Reindex repository",
description = "Rebuilds all transactions and balances from archived blocks. Warning: takes around 1 week, and the core will not function normally during this time. If 'false' is returned, the database may be left in an inconsistent state, requiring another reindex or a bootstrap to correct it.",
responses = {
@ApiResponse(
description = "\"true\"",
content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE, ApiError.BLOCKCHAIN_NEEDS_SYNC})
@SecurityRequirement(name = "apiKey")
public String reindex(@HeaderParam(Security.API_KEY_HEADER) String apiKey) {
Security.checkApiCallAllowed(request);
if (Synchronizer.getInstance().isSynchronizing())
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCKCHAIN_NEEDS_SYNC);
try {
ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
blockchainLock.lockInterruptibly();
try {
ReindexManager reindexManager = new ReindexManager();
reindexManager.reindex();
return "true";
} catch (DataException e) {
LOGGER.info("DataException when reindexing: {}", e.getMessage());
} finally {
blockchainLock.unlock();
}
} catch (InterruptedException e) {
// We couldn't lock blockchain to perform reindex
return "false";
}
return "false";
}
@DELETE @DELETE
@Path("/repository") @Path("/repository")
@Operation( @Operation(
@ -1014,6 +966,8 @@ public class AdminResource {
} }
} }
@POST @POST
@Path("/apikey/generate") @Path("/apikey/generate")
@Operation( @Operation(
@ -1067,29 +1021,4 @@ public class AdminResource {
return "true"; return "true";
} }
@GET
@Path("/systeminfo")
@Operation(
summary = "System Information",
description = "System memory usage and available processors.",
responses = {
@ApiResponse(
description = "memory usage and available processors",
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = SystemInfo.class))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public SystemInfo getSystemInformation() {
SystemInfo info
= new SystemInfo(
Runtime.getRuntime().freeMemory(),
Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(),
Runtime.getRuntime().totalMemory(),
Runtime.getRuntime().maxMemory(),
Runtime.getRuntime().availableProcessors());
return info;
}
} }

View File

@ -71,33 +71,33 @@ public class RenderResource {
@Path("/signature/{signature}") @Path("/signature/{signature}")
@SecurityRequirement(name = "apiKey") @SecurityRequirement(name = "apiKey")
public HttpServletResponse getIndexBySignature(@PathParam("signature") String signature, public HttpServletResponse getIndexBySignature(@PathParam("signature") String signature,
@QueryParam("theme") String theme, @QueryParam("lang") String lang) { @QueryParam("theme") String theme) {
if (!Settings.getInstance().isQDNAuthBypassEnabled()) if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null); Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme, lang); return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme);
} }
@GET @GET
@Path("/signature/{signature}/{path:.*}") @Path("/signature/{signature}/{path:.*}")
@SecurityRequirement(name = "apiKey") @SecurityRequirement(name = "apiKey")
public HttpServletResponse getPathBySignature(@PathParam("signature") String signature, @PathParam("path") String inPath, public HttpServletResponse getPathBySignature(@PathParam("signature") String signature, @PathParam("path") String inPath,
@QueryParam("theme") String theme, @QueryParam("lang") String lang) { @QueryParam("theme") String theme) {
if (!Settings.getInstance().isQDNAuthBypassEnabled()) if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null); Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme, lang); return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme);
} }
@GET @GET
@Path("/hash/{hash}") @Path("/hash/{hash}")
@SecurityRequirement(name = "apiKey") @SecurityRequirement(name = "apiKey")
public HttpServletResponse getIndexByHash(@PathParam("hash") String hash58, @QueryParam("secret") String secret58, public HttpServletResponse getIndexByHash(@PathParam("hash") String hash58, @QueryParam("secret") String secret58,
@QueryParam("theme") String theme, @QueryParam("lang") String lang) { @QueryParam("theme") String theme) {
if (!Settings.getInstance().isQDNAuthBypassEnabled()) if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null); Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme, lang); return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme);
} }
@GET @GET
@ -105,11 +105,11 @@ public class RenderResource {
@SecurityRequirement(name = "apiKey") @SecurityRequirement(name = "apiKey")
public HttpServletResponse getPathByHash(@PathParam("hash") String hash58, @PathParam("path") String inPath, public HttpServletResponse getPathByHash(@PathParam("hash") String hash58, @PathParam("path") String inPath,
@QueryParam("secret") String secret58, @QueryParam("secret") String secret58,
@QueryParam("theme") String theme, @QueryParam("lang") String lang) { @QueryParam("theme") String theme) {
if (!Settings.getInstance().isQDNAuthBypassEnabled()) if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null); Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme, lang); return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme);
} }
@GET @GET
@ -119,12 +119,12 @@ public class RenderResource {
@PathParam("name") String name, @PathParam("name") String name,
@PathParam("path") String inPath, @PathParam("path") String inPath,
@QueryParam("identifier") String identifier, @QueryParam("identifier") String identifier,
@QueryParam("theme") String theme, @QueryParam("lang") String lang) { @QueryParam("theme") String theme) {
if (!Settings.getInstance().isQDNAuthBypassEnabled()) if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, name, service, null); Security.requirePriorAuthorization(request, name, service, null);
String prefix = String.format("/render/%s", service); String prefix = String.format("/render/%s", service);
return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme, lang); return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme);
} }
@GET @GET
@ -133,18 +133,18 @@ public class RenderResource {
public HttpServletResponse getIndexByName(@PathParam("service") Service service, public HttpServletResponse getIndexByName(@PathParam("service") Service service,
@PathParam("name") String name, @PathParam("name") String name,
@QueryParam("identifier") String identifier, @QueryParam("identifier") String identifier,
@QueryParam("theme") String theme, @QueryParam("lang") String lang) { @QueryParam("theme") String theme) {
if (!Settings.getInstance().isQDNAuthBypassEnabled()) if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, name, service, null); Security.requirePriorAuthorization(request, name, service, null);
String prefix = String.format("/render/%s", service); String prefix = String.format("/render/%s", service);
return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme, lang); return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme);
} }
private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier, private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String theme, String lang) { String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String theme) {
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath, ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
secret58, prefix, includeResourceIdInPrefix, async, "render", request, response, context); secret58, prefix, includeResourceIdInPrefix, async, "render", request, response, context);
@ -152,9 +152,6 @@ public class RenderResource {
if (theme != null) { if (theme != null) {
renderer.setTheme(theme); renderer.setTheme(theme);
} }
if (lang != null) {
renderer.setLang(lang);
}
return renderer.render(); return renderer.render();
} }

View File

@ -77,9 +77,7 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
} }
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
Boolean hasChatReference = getHasChatReference(session); ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session));
ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session), hasChatReference);
StringWriter stringWriter = new StringWriter(); StringWriter stringWriter = new StringWriter();
@ -105,20 +103,4 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
return Encoding.valueOf(encoding); return Encoding.valueOf(encoding);
} }
private Boolean getHasChatReference(Session session) {
Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
List<String> hasChatReferenceList = queryParams.get("haschatreference");
// Return null if not specified
if (hasChatReferenceList != null && hasChatReferenceList.size() == 1) {
String value = hasChatReferenceList.get(0).toLowerCase();
if (value.equals("true")) {
return true;
} else if (value.equals("false")) {
return false;
}
}
return null; // Ignored if not present
}
} }

View File

@ -1,102 +0,0 @@
package org.qortal.api.websocket;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.eclipse.jetty.websocket.api.Session;
import org.eclipse.jetty.websocket.api.WebSocketException;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage;
import org.eclipse.jetty.websocket.api.annotations.WebSocket;
import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
import org.qortal.api.ApiError;
import org.qortal.controller.Controller;
import org.qortal.data.arbitrary.DataMonitorInfo;
import org.qortal.event.DataMonitorEvent;
import org.qortal.event.Event;
import org.qortal.event.EventBus;
import org.qortal.event.Listener;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.utils.Base58;
import java.io.IOException;
import java.io.StringWriter;
import java.util.List;
@WebSocket
@SuppressWarnings("serial")
public class DataMonitorSocket extends ApiWebSocket implements Listener {
private static final Logger LOGGER = LogManager.getLogger(DataMonitorSocket.class);
@Override
public void configure(WebSocketServletFactory factory) {
LOGGER.info("configure");
factory.register(DataMonitorSocket.class);
EventBus.INSTANCE.addListener(this);
}
@Override
public void listen(Event event) {
if (!(event instanceof DataMonitorEvent))
return;
DataMonitorEvent dataMonitorEvent = (DataMonitorEvent) event;
for (Session session : getSessions())
sendDataEventSummary(session, buildInfo(dataMonitorEvent));
}
private DataMonitorInfo buildInfo(DataMonitorEvent dataMonitorEvent) {
return new DataMonitorInfo(
dataMonitorEvent.getTimestamp(),
dataMonitorEvent.getIdentifier(),
dataMonitorEvent.getName(),
dataMonitorEvent.getService(),
dataMonitorEvent.getDescription(),
dataMonitorEvent.getTransactionTimestamp(),
dataMonitorEvent.getLatestPutTimestamp()
);
}
@OnWebSocketConnect
@Override
public void onWebSocketConnect(Session session) {
super.onWebSocketConnect(session);
}
@OnWebSocketClose
@Override
public void onWebSocketClose(Session session, int statusCode, String reason) {
super.onWebSocketClose(session, statusCode, reason);
}
@OnWebSocketError
public void onWebSocketError(Session session, Throwable throwable) {
/* We ignore errors for now, but method here to silence log spam */
}
@OnWebSocketMessage
public void onWebSocketMessage(Session session, String message) {
LOGGER.info("onWebSocketMessage: message = " + message);
}
private void sendDataEventSummary(Session session, DataMonitorInfo dataMonitorInfo) {
StringWriter stringWriter = new StringWriter();
try {
marshall(stringWriter, dataMonitorInfo);
session.getRemote().sendStringByFuture(stringWriter.toString());
} catch (IOException | WebSocketException e) {
// No output this time
}
}
}

View File

@ -98,7 +98,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
byte[] codeHash = acctInfo.getKey().value; byte[] codeHash = acctInfo.getKey().value;
ACCT acct = acctInfo.getValue().get(); ACCT acct = acctInfo.getValue().get();
List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash, null, null, List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
isFinished, dataByteOffset, expectedValue, minimumFinalHeight, isFinished, dataByteOffset, expectedValue, minimumFinalHeight,
null, null, null); null, null, null);
@ -259,7 +259,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
ACCT acct = acctInfo.getValue().get(); ACCT acct = acctInfo.getValue().get();
Integer dataByteOffset = acct.getModeByteOffset(); Integer dataByteOffset = acct.getModeByteOffset();
List<ATStateData> initialAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash, null, null, List<ATStateData> initialAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
isFinished, dataByteOffset, expectedValue, minimumFinalHeight, isFinished, dataByteOffset, expectedValue, minimumFinalHeight,
null, null, null); null, null, null);
@ -298,7 +298,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
byte[] codeHash = acctInfo.getKey().value; byte[] codeHash = acctInfo.getKey().value;
ACCT acct = acctInfo.getValue().get(); ACCT acct = acctInfo.getValue().get();
List<ATStateData> historicAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash, null, null, List<ATStateData> historicAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
isFinished, dataByteOffset, expectedValue, minimumFinalHeight, isFinished, dataByteOffset, expectedValue, minimumFinalHeight,
null, null, null); null, null, null);

View File

@ -1,83 +0,0 @@
package org.qortal.api.websocket;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.eclipse.jetty.websocket.api.Session;
import org.eclipse.jetty.websocket.api.WebSocketException;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage;
import org.eclipse.jetty.websocket.api.annotations.WebSocket;
import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
import org.qortal.data.crosschain.UnsignedFeeEvent;
import org.qortal.event.Event;
import org.qortal.event.EventBus;
import org.qortal.event.FeeWaitingEvent;
import org.qortal.event.Listener;
import java.io.IOException;
import java.io.StringWriter;
@WebSocket
@SuppressWarnings("serial")
public class UnsignedFeesSocket extends ApiWebSocket implements Listener {
private static final Logger LOGGER = LogManager.getLogger(UnsignedFeesSocket.class);
@Override
public void configure(WebSocketServletFactory factory) {
LOGGER.info("configure");
factory.register(UnsignedFeesSocket.class);
EventBus.INSTANCE.addListener(this);
}
@Override
public void listen(Event event) {
if (!(event instanceof FeeWaitingEvent))
return;
for (Session session : getSessions()) {
FeeWaitingEvent feeWaitingEvent = (FeeWaitingEvent) event;
sendUnsignedFeeEvent(session, new UnsignedFeeEvent(feeWaitingEvent.isPositive(), feeWaitingEvent.getAddress()));
}
}
@OnWebSocketConnect
@Override
public void onWebSocketConnect(Session session) {
super.onWebSocketConnect(session);
}
@OnWebSocketClose
@Override
public void onWebSocketClose(Session session, int statusCode, String reason) {
super.onWebSocketClose(session, statusCode, reason);
}
@OnWebSocketError
public void onWebSocketError(Session session, Throwable throwable) {
/* We ignore errors for now, but method here to silence log spam */
}
@OnWebSocketMessage
public void onWebSocketMessage(Session session, String message) {
LOGGER.info("onWebSocketMessage: message = " + message);
}
private void sendUnsignedFeeEvent(Session session, UnsignedFeeEvent unsignedFeeEvent) {
StringWriter stringWriter = new StringWriter();
try {
marshall(stringWriter, unsignedFeeEvent);
session.getRemote().sendStringByFuture(stringWriter.toString());
} catch (IOException | WebSocketException e) {
// No output this time
}
}
}

View File

@ -4,12 +4,9 @@ import org.qortal.repository.DataException;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import java.io.IOException; import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystem;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.MessageDigest; import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException; import java.security.NoSuchAlgorithmException;
import java.util.Arrays; import java.util.Arrays;
@ -26,11 +23,7 @@ public class ArbitraryDataDigest {
} }
public void compute() throws IOException, DataException { public void compute() throws IOException, DataException {
List<Path> allPaths = Files.walk(path) List<Path> allPaths = Files.walk(path).filter(Files::isRegularFile).sorted().collect(Collectors.toList());
.filter(Files::isRegularFile)
.sorted()
.collect(Collectors.toList());
Path basePathAbsolute = this.path.toAbsolutePath(); Path basePathAbsolute = this.path.toAbsolutePath();
MessageDigest sha256; MessageDigest sha256;
@ -51,29 +44,17 @@ public class ArbitraryDataDigest {
continue; continue;
} }
// Account for \ VS / : Linux VS Windows
String pathString = relativePath.toString();
if (relativePath.getFileSystem().toString().contains("Windows")) {
pathString = pathString.replace("\\", "/");
}
// Hash path // Hash path
byte[] filePathBytes = pathString.getBytes(StandardCharsets.UTF_8); byte[] filePathBytes = relativePath.toString().getBytes(StandardCharsets.UTF_8);
sha256.update(filePathBytes); sha256.update(filePathBytes);
try (InputStream in = Files.newInputStream(path)) { // Hash contents
byte[] buffer = new byte[65536]; // 64 KB byte[] fileContent = Files.readAllBytes(path);
int bytesRead; sha256.update(fileContent);
while ((bytesRead = in.read(buffer)) != -1) {
sha256.update(buffer, 0, bytesRead);
}
}
} }
this.hash = sha256.digest(); this.hash = sha256.digest();
} }
public boolean isHashValid(byte[] hash) { public boolean isHashValid(byte[] hash) {
return Arrays.equals(hash, this.hash); return Arrays.equals(hash, this.hash);
} }

View File

@ -52,7 +52,7 @@ public class ArbitraryDataFile {
private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFile.class); private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFile.class);
public static final long MAX_FILE_SIZE = 2L * 1024 * 1024 * 1024; // 2 GiB public static final long MAX_FILE_SIZE = 500 * 1024 * 1024; // 500MiB
protected static final int MAX_CHUNK_SIZE = 1 * 1024 * 1024; // 1MiB protected static final int MAX_CHUNK_SIZE = 1 * 1024 * 1024; // 1MiB
public static final int CHUNK_SIZE = 512 * 1024; // 0.5MiB public static final int CHUNK_SIZE = 512 * 1024; // 0.5MiB
public static int SHORT_DIGEST_LENGTH = 8; public static int SHORT_DIGEST_LENGTH = 8;

View File

@ -439,15 +439,7 @@ public class ArbitraryDataReader {
// Ensure the complete hash matches the joined chunks // Ensure the complete hash matches the joined chunks
if (!Arrays.equals(arbitraryDataFile.digest(), transactionData.getData())) { if (!Arrays.equals(arbitraryDataFile.digest(), transactionData.getData())) {
// Delete the invalid file // Delete the invalid file
LOGGER.info("Deleting invalid file: path = " + arbitraryDataFile.getFilePath()); arbitraryDataFile.delete();
if( arbitraryDataFile.delete() ) {
LOGGER.info("Deleted invalid file successfully: path = " + arbitraryDataFile.getFilePath());
}
else {
LOGGER.warn("Could not delete invalid file: path = " + arbitraryDataFile.getFilePath());
}
throw new DataException("Unable to validate complete file hash"); throw new DataException("Unable to validate complete file hash");
} }
} }

View File

@ -1,7 +1,6 @@
package org.qortal.arbitrary; package org.qortal.arbitrary;
import com.google.common.io.Resources; import com.google.common.io.Resources;
import org.apache.commons.io.FileUtils; import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.ArrayUtils; import org.apache.commons.lang3.ArrayUtils;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
@ -16,13 +15,11 @@ import org.qortal.settings.Settings;
import javax.servlet.ServletContext; import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import javax.servlet.http.HttpServletResponse;
import java.io.File; import java.io.File;
import java.io.FileInputStream; import java.io.FileInputStream;
import java.io.FileNotFoundException; import java.io.FileNotFoundException;
import java.io.IOException; import java.io.IOException;
import java.net.URL; import java.net.URL;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.NoSuchFileException; import java.nio.file.NoSuchFileException;
@ -40,7 +37,6 @@ public class ArbitraryDataRenderer {
private final Service service; private final Service service;
private final String identifier; private final String identifier;
private String theme = "light"; private String theme = "light";
private String lang = "en";
private String inPath; private String inPath;
private final String secret58; private final String secret58;
private final String prefix; private final String prefix;
@ -170,16 +166,9 @@ public class ArbitraryDataRenderer {
if (HTMLParser.isHtmlFile(filename)) { if (HTMLParser.isHtmlFile(filename)) {
// HTML file - needs to be parsed // HTML file - needs to be parsed
byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory
String encodedResourceId; HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, includeResourceIdInPrefix, data, qdnContext, service, identifier, theme, usingCustomRouting);
if (resourceIdType == ResourceIdType.NAME) {
encodedResourceId = resourceId.replace(" ", "%20");
} else {
encodedResourceId = resourceId;
}
HTMLParser htmlParser = new HTMLParser(encodedResourceId, inPath, prefix, includeResourceIdInPrefix, data, qdnContext, service, identifier, theme, usingCustomRouting, lang);
htmlParser.addAdditionalHeaderTags(); htmlParser.addAdditionalHeaderTags();
response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; font-src 'self' data:; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' wss: blob:;"); response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:;");
response.setContentType(context.getMimeType(filename)); response.setContentType(context.getMimeType(filename));
response.setContentLength(htmlParser.getData().length); response.setContentLength(htmlParser.getData().length);
response.getOutputStream().write(htmlParser.getData()); response.getOutputStream().write(htmlParser.getData());
@ -267,8 +256,5 @@ public class ArbitraryDataRenderer {
public void setTheme(String theme) { public void setTheme(String theme) {
this.theme = theme; this.theme = theme;
} }
public void setLang(String lang) {
this.lang = lang;
}
} }

View File

@ -29,7 +29,6 @@ import org.qortal.utils.FilesystemUtils;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import java.io.IOException; import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
@ -198,7 +197,7 @@ public class ArbitraryDataTransactionBuilder {
// We can't use PATCH for on-chain data because this requires the .qortal directory, which can't be put on chain // We can't use PATCH for on-chain data because this requires the .qortal directory, which can't be put on chain
final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(this.path, false); final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(this.path, false);
final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(Files.size(path)) <= ArbitraryTransaction.MAX_DATA_SIZE); final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
if (shouldUseOnChainData) { if (shouldUseOnChainData) {
LOGGER.info("Data size is small enough to go on chain - using PUT"); LOGGER.info("Data size is small enough to go on chain - using PUT");
return Method.PUT; return Method.PUT;
@ -246,7 +245,7 @@ public class ArbitraryDataTransactionBuilder {
// Single file resources are handled differently, especially for very small data payloads, as these go on chain // Single file resources are handled differently, especially for very small data payloads, as these go on chain
final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(path, false); final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(path, false);
final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(Files.size(path)) <= ArbitraryTransaction.MAX_DATA_SIZE); final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
// Use zip compression if data isn't going on chain // Use zip compression if data isn't going on chain
Compression compression = shouldUseOnChainData ? Compression.NONE : Compression.ZIP; Compression compression = shouldUseOnChainData ? Compression.NONE : Compression.ZIP;

View File

@ -37,7 +37,7 @@ public enum Service {
if (files != null && files[0] != null) { if (files != null && files[0] != null) {
final String extension = FilenameUtils.getExtension(files[0].getName()).toLowerCase(); final String extension = FilenameUtils.getExtension(files[0].getName()).toLowerCase();
// We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string // We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string
final List<String> allowedExtensions = Arrays.asList("qortal", "zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", ""); final List<String> allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "");
if (extension == null || !allowedExtensions.contains(extension)) { if (extension == null || !allowedExtensions.contains(extension)) {
return ValidationResult.INVALID_FILE_EXTENSION; return ValidationResult.INVALID_FILE_EXTENSION;
} }
@ -62,17 +62,7 @@ public enum Service {
// Custom validation function to require an index HTML file in the root directory // Custom validation function to require an index HTML file in the root directory
List<String> fileNames = ArbitraryDataRenderer.indexFiles(); List<String> fileNames = ArbitraryDataRenderer.indexFiles();
List<String> files; String[] files = path.toFile().list();
// single files are paackaged differently
if( path.toFile().isFile() ) {
files = new ArrayList<>(1);
files.add(path.getFileName().toString());
}
else {
files = new ArrayList<>(Arrays.asList(path.toFile().list()));
}
if (files != null) { if (files != null) {
for (String file : files) { for (String file : files) {
Path fileName = Paths.get(file).getFileName(); Path fileName = Paths.get(file).getFileName();
@ -177,7 +167,7 @@ public enum Service {
COMMENT(1800, true, 500*1024L, true, false, null), COMMENT(1800, true, 500*1024L, true, false, null),
CHAIN_COMMENT(1810, true, 239L, true, false, null), CHAIN_COMMENT(1810, true, 239L, true, false, null),
MAIL(1900, true, 1024*1024L, true, false, null), MAIL(1900, true, 1024*1024L, true, false, null),
MAIL_PRIVATE(1901, true, 5*1024*1024L, true, true, null), MAIL_PRIVATE(1901, true, 1024*1024L, true, true, null),
MESSAGE(1910, true, 1024*1024L, true, false, null), MESSAGE(1910, true, 1024*1024L, true, false, null),
MESSAGE_PRIVATE(1911, true, 1024*1024L, true, true, null); MESSAGE_PRIVATE(1911, true, 1024*1024L, true, true, null);

View File

@ -23,12 +23,12 @@ import org.qortal.data.at.ATStateData;
import org.qortal.data.block.BlockData; import org.qortal.data.block.BlockData;
import org.qortal.data.block.BlockSummaryData; import org.qortal.data.block.BlockSummaryData;
import org.qortal.data.block.BlockTransactionData; import org.qortal.data.block.BlockTransactionData;
import org.qortal.data.group.GroupAdminData;
import org.qortal.data.network.OnlineAccountData; import org.qortal.data.network.OnlineAccountData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
import org.qortal.group.Group; import org.qortal.repository.ATRepository;
import org.qortal.repository.*; import org.qortal.repository.DataException;
import org.qortal.settings.Settings; import org.qortal.repository.Repository;
import org.qortal.repository.TransactionRepository;
import org.qortal.transaction.AtTransaction; import org.qortal.transaction.AtTransaction;
import org.qortal.transaction.Transaction; import org.qortal.transaction.Transaction;
import org.qortal.transaction.Transaction.ApprovalStatus; import org.qortal.transaction.Transaction.ApprovalStatus;
@ -39,7 +39,6 @@ import org.qortal.transform.block.BlockTransformer;
import org.qortal.transform.transaction.TransactionTransformer; import org.qortal.transform.transaction.TransactionTransformer;
import org.qortal.utils.Amounts; import org.qortal.utils.Amounts;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import org.qortal.utils.Groups;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import java.io.ByteArrayOutputStream; import java.io.ByteArrayOutputStream;
@ -105,7 +104,6 @@ public class Block {
protected Repository repository; protected Repository repository;
protected BlockData blockData; protected BlockData blockData;
protected PublicKeyAccount minter; protected PublicKeyAccount minter;
boolean isTestnet = Settings.getInstance().isTestNet();
// Other properties // Other properties
private static final Logger LOGGER = LogManager.getLogger(Block.class); private static final Logger LOGGER = LogManager.getLogger(Block.class);
@ -144,14 +142,11 @@ public class Block {
private final Account mintingAccount; private final Account mintingAccount;
private final AccountData mintingAccountData; private final AccountData mintingAccountData;
private final boolean isMinterFounder; private final boolean isMinterFounder;
private final boolean isMinterMember;
private final Account recipientAccount; private final Account recipientAccount;
private final AccountData recipientAccountData; private final AccountData recipientAccountData;
final BlockChain blockChain = BlockChain.getInstance(); ExpandedAccount(Repository repository, RewardShareData rewardShareData) throws DataException {
ExpandedAccount(Repository repository, RewardShareData rewardShareData, int blockHeight) throws DataException {
this.rewardShareData = rewardShareData; this.rewardShareData = rewardShareData;
this.sharePercent = this.rewardShareData.getSharePercent(); this.sharePercent = this.rewardShareData.getSharePercent();
@ -160,12 +155,6 @@ public class Block {
this.isMinterFounder = Account.isFounder(mintingAccountData.getFlags()); this.isMinterFounder = Account.isFounder(mintingAccountData.getFlags());
this.isRecipientAlsoMinter = this.rewardShareData.getRecipient().equals(this.mintingAccount.getAddress()); this.isRecipientAlsoMinter = this.rewardShareData.getRecipient().equals(this.mintingAccount.getAddress());
this.isMinterMember
= Groups.memberExistsInAnyGroup(
repository.getGroupRepository(),
Groups.getGroupIdsToMint(BlockChain.getInstance(), blockHeight),
this.mintingAccount.getAddress()
);
if (this.isRecipientAlsoMinter) { if (this.isRecipientAlsoMinter) {
// Self-share: minter is also recipient // Self-share: minter is also recipient
@ -178,19 +167,6 @@ public class Block {
} }
} }
/**
* Get Effective Minting Level
*
* @return the effective minting level, if a data exception is thrown, it catches the exception and returns a zero
*/
public int getEffectiveMintingLevel() {
try {
return this.mintingAccount.getEffectiveMintingLevel();
} catch (DataException e) {
return 0;
}
}
public Account getMintingAccount() { public Account getMintingAccount() {
return this.mintingAccount; return this.mintingAccount;
} }
@ -207,19 +183,15 @@ public class Block {
* @return account-level share "bin" from blockchain config, or null if founder / none found * @return account-level share "bin" from blockchain config, or null if founder / none found
*/ */
public AccountLevelShareBin getShareBin(int blockHeight) { public AccountLevelShareBin getShareBin(int blockHeight) {
if (this.isMinterFounder && blockHeight < BlockChain.getInstance().getAdminsReplaceFoundersHeight()) if (this.isMinterFounder)
return null; return null;
final int accountLevel = this.mintingAccountData.getLevel(); final int accountLevel = this.mintingAccountData.getLevel();
if (accountLevel <= 0) if (accountLevel <= 0)
return null; // level 0 isn't included in any share bins return null; // level 0 isn't included in any share bins
if (blockHeight >= blockChain.getFixBatchRewardHeight()) {
if (!this.isMinterMember)
return null; // not member of minter group isn't included in any share bins
}
// Select the correct set of share bins based on block height // Select the correct set of share bins based on block height
final BlockChain blockChain = BlockChain.getInstance();
final AccountLevelShareBin[] shareBinsByLevel = (blockHeight >= blockChain.getSharesByLevelV2Height()) ? final AccountLevelShareBin[] shareBinsByLevel = (blockHeight >= blockChain.getSharesByLevelV2Height()) ?
blockChain.getShareBinsByAccountLevelV2() : blockChain.getShareBinsByAccountLevelV1(); blockChain.getShareBinsByAccountLevelV2() : blockChain.getShareBinsByAccountLevelV1();
@ -424,9 +396,7 @@ public class Block {
onlineAccounts.removeIf(a -> a.getNonce() == null || a.getNonce() < 0); onlineAccounts.removeIf(a -> a.getNonce() == null || a.getNonce() < 0);
// After feature trigger, remove any online accounts that are level 0 // After feature trigger, remove any online accounts that are level 0
// but only if they are before the ignore level feature trigger if (height >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
if (height < BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() &&
height >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
onlineAccounts.removeIf(a -> { onlineAccounts.removeIf(a -> {
try { try {
return Account.getRewardShareEffectiveMintingLevel(repository, a.getPublicKey()) == 0; return Account.getRewardShareEffectiveMintingLevel(repository, a.getPublicKey()) == 0;
@ -437,21 +407,6 @@ public class Block {
}); });
} }
// After feature trigger, remove any online accounts that are not minter group member
if (height >= BlockChain.getInstance().getGroupMemberCheckHeight()) {
onlineAccounts.removeIf(a -> {
try {
List<Integer> groupIdsToMint = Groups.getGroupIdsToMint(BlockChain.getInstance(), height);
String address = Account.getRewardShareMintingAddress(repository, a.getPublicKey());
boolean isMinterGroupMember = Groups.memberExistsInAnyGroup(repository.getGroupRepository(), groupIdsToMint, address);
return !isMinterGroupMember;
} catch (DataException e) {
// Something went wrong, so remove the account
return true;
}
});
}
if (onlineAccounts.isEmpty()) { if (onlineAccounts.isEmpty()) {
LOGGER.debug("No online accounts - not even our own?"); LOGGER.debug("No online accounts - not even our own?");
return null; return null;
@ -758,12 +713,10 @@ public class Block {
List<ExpandedAccount> expandedAccounts = new ArrayList<>(); List<ExpandedAccount> expandedAccounts = new ArrayList<>();
for (RewardShareData rewardShare : this.cachedOnlineRewardShares) { for (RewardShareData rewardShare : this.cachedOnlineRewardShares)
expandedAccounts.add(new ExpandedAccount(repository, rewardShare, this.blockData.getHeight())); expandedAccounts.add(new ExpandedAccount(repository, rewardShare));
}
this.cachedExpandedAccounts = expandedAccounts; this.cachedExpandedAccounts = expandedAccounts;
LOGGER.trace(() -> String.format("Online reward-shares after expanded accounts %s", this.cachedOnlineRewardShares));
return this.cachedExpandedAccounts; return this.cachedExpandedAccounts;
} }
@ -1169,32 +1122,14 @@ public class Block {
if (onlineRewardShares == null) if (onlineRewardShares == null)
return ValidationResult.ONLINE_ACCOUNT_UNKNOWN; return ValidationResult.ONLINE_ACCOUNT_UNKNOWN;
// After feature trigger, require all online account minters to be greater than level 0, // After feature trigger, require all online account minters to be greater than level 0
// but only if it is before the feature trigger where we ignore level again if (this.getBlockData().getHeight() >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
if (this.blockData.getHeight() < BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() && List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
this.getBlockData().getHeight() >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
List<ExpandedAccount> expandedAccounts
= this.getExpandedAccounts().stream()
.filter(expandedAccount -> expandedAccount.isMinterMember)
.collect(Collectors.toList());
for (ExpandedAccount account : expandedAccounts) { for (ExpandedAccount account : expandedAccounts) {
if (account.getMintingAccount().getEffectiveMintingLevel() == 0) if (account.getMintingAccount().getEffectiveMintingLevel() == 0)
return ValidationResult.ONLINE_ACCOUNTS_INVALID; return ValidationResult.ONLINE_ACCOUNTS_INVALID;
if (this.getBlockData().getHeight() >= BlockChain.getInstance().getFixBatchRewardHeight()) {
if (!account.isMinterMember)
return ValidationResult.ONLINE_ACCOUNTS_INVALID;
}
} }
} }
else if (this.blockData.getHeight() >= BlockChain.getInstance().getIgnoreLevelForRewardShareHeight()){
Optional<ExpandedAccount> anyInvalidAccount
= this.getExpandedAccounts().stream()
.filter(account -> !account.isMinterMember)
.findAny();
if( anyInvalidAccount.isPresent() ) return ValidationResult.ONLINE_ACCOUNTS_INVALID;
}
// If block is past a certain age then we simply assume the signatures were correct // If block is past a certain age then we simply assume the signatures were correct
long signatureRequirementThreshold = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMinLifetime(); long signatureRequirementThreshold = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMinLifetime();
@ -1321,7 +1256,6 @@ public class Block {
// Online Accounts // Online Accounts
ValidationResult onlineAccountsResult = this.areOnlineAccountsValid(); ValidationResult onlineAccountsResult = this.areOnlineAccountsValid();
LOGGER.trace("Accounts valid = {}", onlineAccountsResult);
if (onlineAccountsResult != ValidationResult.OK) if (onlineAccountsResult != ValidationResult.OK)
return onlineAccountsResult; return onlineAccountsResult;
@ -1347,20 +1281,13 @@ public class Block {
// Create repository savepoint here so we can rollback to it after testing transactions // Create repository savepoint here so we can rollback to it after testing transactions
repository.setSavepoint(); repository.setSavepoint();
if (!isTestnet) { if (this.blockData.getHeight() == 212937) {
if (this.blockData.getHeight() == 212937) { // Apply fix for block 212937 but fix will be rolled back before we exit method
// Apply fix for block 212937 but fix will be rolled back before we exit method Block212937.processFix(this);
Block212937.processFix(this); }
} else if (this.blockData.getHeight() == 1333492) { else if (InvalidNameRegistrationBlocks.isAffectedBlock(this.blockData.getHeight())) {
// Apply fix for block 1333492 but fix will be rolled back before we exit method // Apply fix for affected name registration blocks, but fix will be rolled back before we exit method
Block1333492.processFix(this); InvalidNameRegistrationBlocks.processFix(this);
} else if (InvalidNameRegistrationBlocks.isAffectedBlock(this.blockData.getHeight())) {
// Apply fix for affected name registration blocks, but fix will be rolled back before we exit method
InvalidNameRegistrationBlocks.processFix(this);
} else if (InvalidBalanceBlocks.isAffectedBlock(this.blockData.getHeight())) {
// Apply fix for affected balance blocks, but fix will be rolled back before we exit method
InvalidBalanceBlocks.processFix(this);
}
} }
for (Transaction transaction : this.getTransactions()) { for (Transaction transaction : this.getTransactions()) {
@ -1410,7 +1337,7 @@ public class Block {
// Check transaction can even be processed // Check transaction can even be processed
validationResult = transaction.isProcessable(); validationResult = transaction.isProcessable();
if (validationResult != Transaction.ValidationResult.OK) { if (validationResult != Transaction.ValidationResult.OK) {
LOGGER.debug(String.format("Error during transaction validation, tx %s: %s", Base58.encode(transactionData.getSignature()), validationResult.name())); LOGGER.info(String.format("Error during transaction validation, tx %s: %s", Base58.encode(transactionData.getSignature()), validationResult.name()));
return ValidationResult.TRANSACTION_INVALID; return ValidationResult.TRANSACTION_INVALID;
} }
@ -1582,7 +1509,7 @@ public class Block {
return false; return false;
Account mintingAccount = new PublicKeyAccount(this.repository, rewardShareData.getMinterPublicKey()); Account mintingAccount = new PublicKeyAccount(this.repository, rewardShareData.getMinterPublicKey());
return mintingAccount.canMint(false); return mintingAccount.canMint();
} }
/** /**
@ -1611,7 +1538,6 @@ public class Block {
this.blockData.setHeight(blockchainHeight + 1); this.blockData.setHeight(blockchainHeight + 1);
LOGGER.trace(() -> String.format("Processing block %d", this.blockData.getHeight())); LOGGER.trace(() -> String.format("Processing block %d", this.blockData.getHeight()));
LOGGER.trace(() -> String.format("Online Reward Shares in process %s", this.cachedOnlineRewardShares));
if (this.blockData.getHeight() > 1) { if (this.blockData.getHeight() > 1) {
@ -1624,25 +1550,21 @@ public class Block {
processBlockRewards(); processBlockRewards();
} }
if (!isTestnet) { if (this.blockData.getHeight() == 212937) {
if (this.blockData.getHeight() == 212937) { // Apply fix for block 212937
// Apply fix for block 212937 Block212937.processFix(this);
Block212937.processFix(this); }
} else if (this.blockData.getHeight() == 1333492) {
// Apply fix for block 1333492 if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height()) {
Block1333492.processFix(this); SelfSponsorshipAlgoV1Block.processAccountPenalties(this);
} else if (InvalidBalanceBlocks.isAffectedBlock(this.blockData.getHeight())) { }
// Apply fix for affected balance blocks
InvalidBalanceBlocks.processFix(this); if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV2Height()) {
} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height()) { SelfSponsorshipAlgoV2Block.processAccountPenalties(this);
SelfSponsorshipAlgoV1Block.processAccountPenalties(this); }
} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV2Height()) {
SelfSponsorshipAlgoV2Block.processAccountPenalties(this); if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV3Height()) {
} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV3Height()) { SelfSponsorshipAlgoV3Block.processAccountPenalties(this);
SelfSponsorshipAlgoV3Block.processAccountPenalties(this);
} else if (this.blockData.getHeight() == BlockChain.getInstance().getMultipleNamesPerAccountHeight()) {
PrimaryNamesBlock.processNames(this.repository);
}
} }
} }
@ -1685,17 +1607,7 @@ public class Block {
final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel(); final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel();
final int maximumLevel = cumulativeBlocksByLevel.size() - 1; final int maximumLevel = cumulativeBlocksByLevel.size() - 1;
final List<ExpandedAccount> expandedAccounts; final List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
if (this.getBlockData().getHeight() < BlockChain.getInstance().getFixBatchRewardHeight()) {
expandedAccounts = this.getExpandedAccounts().stream().collect(Collectors.toList());
}
else {
expandedAccounts
= this.getExpandedAccounts().stream()
.filter(expandedAccount -> expandedAccount.isMinterMember)
.collect(Collectors.toList());
}
Set<AccountData> allUniqueExpandedAccounts = new HashSet<>(); Set<AccountData> allUniqueExpandedAccounts = new HashSet<>();
for (ExpandedAccount expandedAccount : expandedAccounts) { for (ExpandedAccount expandedAccount : expandedAccounts) {
@ -1723,19 +1635,11 @@ public class Block {
accountData.setBlocksMinted(accountData.getBlocksMinted() + 1); accountData.setBlocksMinted(accountData.getBlocksMinted() + 1);
LOGGER.trace(() -> String.format("Block minter %s up to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : ""))); LOGGER.trace(() -> String.format("Block minter %s up to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : "")));
int blocksMintedAdjustment final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
=
(this.blockData.getHeight() > BlockChain.getInstance().getMintedBlocksAdjustmentRemovalHeight())
?
0
:
accountData.getBlocksMintedAdjustment();
final int effectiveBlocksMinted = accountData.getBlocksMinted() + blocksMintedAdjustment + accountData.getBlocksMintedPenalty();
for (int newLevel = maximumLevel; newLevel >= 0; --newLevel) for (int newLevel = maximumLevel; newLevel >= 0; --newLevel)
if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) { if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {
if (newLevel != accountData.getLevel()) { if (newLevel > accountData.getLevel()) {
// Account has increased in level! // Account has increased in level!
accountData.setLevel(newLevel); accountData.setLevel(newLevel);
bumpedAccounts.put(accountData.getAddress(), newLevel); bumpedAccounts.put(accountData.getAddress(), newLevel);
@ -1946,25 +1850,21 @@ public class Block {
// Invalidate expandedAccounts as they may have changed due to orphaning TRANSFER_PRIVS transactions, etc. // Invalidate expandedAccounts as they may have changed due to orphaning TRANSFER_PRIVS transactions, etc.
this.cachedExpandedAccounts = null; this.cachedExpandedAccounts = null;
if (!isTestnet) { if (this.blockData.getHeight() == 212937) {
if (this.blockData.getHeight() == 212937) { // Revert fix for block 212937
// Revert fix for block 212937 Block212937.orphanFix(this);
Block212937.orphanFix(this); }
} else if (this.blockData.getHeight() == 1333492) {
// Revert fix for block 1333492 if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height()) {
Block1333492.orphanFix(this); SelfSponsorshipAlgoV1Block.orphanAccountPenalties(this);
} else if (InvalidBalanceBlocks.isAffectedBlock(this.blockData.getHeight())) { }
// Revert fix for affected balance blocks
InvalidBalanceBlocks.orphanFix(this); if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV2Height()) {
} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height()) { SelfSponsorshipAlgoV2Block.orphanAccountPenalties(this);
SelfSponsorshipAlgoV1Block.orphanAccountPenalties(this); }
} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV2Height()) {
SelfSponsorshipAlgoV2Block.orphanAccountPenalties(this); if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV3Height()) {
} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV3Height()) { SelfSponsorshipAlgoV3Block.orphanAccountPenalties(this);
SelfSponsorshipAlgoV3Block.orphanAccountPenalties(this);
} else if (this.blockData.getHeight() == BlockChain.getInstance().getMultipleNamesPerAccountHeight()) {
PrimaryNamesBlock.orphanNames( this.repository );
}
} }
// Account levels and block rewards are only processed/orphaned on block reward distribution blocks // Account levels and block rewards are only processed/orphaned on block reward distribution blocks
@ -2105,17 +2005,7 @@ public class Block {
final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel(); final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel();
final int maximumLevel = cumulativeBlocksByLevel.size() - 1; final int maximumLevel = cumulativeBlocksByLevel.size() - 1;
final List<ExpandedAccount> expandedAccounts; final List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
if (this.getBlockData().getHeight() < BlockChain.getInstance().getFixBatchRewardHeight()) {
expandedAccounts = this.getExpandedAccounts().stream().collect(Collectors.toList());
}
else {
expandedAccounts
= this.getExpandedAccounts().stream()
.filter(expandedAccount -> expandedAccount.isMinterMember)
.collect(Collectors.toList());
}
Set<AccountData> allUniqueExpandedAccounts = new HashSet<>(); Set<AccountData> allUniqueExpandedAccounts = new HashSet<>();
for (ExpandedAccount expandedAccount : expandedAccounts) { for (ExpandedAccount expandedAccount : expandedAccounts) {
@ -2139,19 +2029,11 @@ public class Block {
accountData.setBlocksMinted(accountData.getBlocksMinted() - 1); accountData.setBlocksMinted(accountData.getBlocksMinted() - 1);
LOGGER.trace(() -> String.format("Block minter %s down to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : ""))); LOGGER.trace(() -> String.format("Block minter %s down to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : "")));
int blocksMintedAdjustment final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
=
(this.blockData.getHeight() -1 > BlockChain.getInstance().getMintedBlocksAdjustmentRemovalHeight())
?
0
:
accountData.getBlocksMintedAdjustment();
final int effectiveBlocksMinted = accountData.getBlocksMinted() + blocksMintedAdjustment + accountData.getBlocksMintedPenalty();
for (int newLevel = maximumLevel; newLevel >= 0; --newLevel) for (int newLevel = maximumLevel; newLevel >= 0; --newLevel)
if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) { if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {
if (newLevel != accountData.getLevel()) { if (newLevel < accountData.getLevel()) {
// Account has decreased in level! // Account has decreased in level!
accountData.setLevel(newLevel); accountData.setLevel(newLevel);
repository.getAccountRepository().setLevel(accountData); repository.getAccountRepository().setLevel(accountData);
@ -2318,7 +2200,6 @@ public class Block {
List<AccountBalanceData> accountBalanceDeltas = balanceChanges.entrySet().stream() List<AccountBalanceData> accountBalanceDeltas = balanceChanges.entrySet().stream()
.map(entry -> new AccountBalanceData(entry.getKey(), Asset.QORT, entry.getValue())) .map(entry -> new AccountBalanceData(entry.getKey(), Asset.QORT, entry.getValue()))
.collect(Collectors.toList()); .collect(Collectors.toList());
LOGGER.trace("Account Balance Deltas: {}", accountBalanceDeltas);
this.repository.getAccountRepository().modifyAssetBalances(accountBalanceDeltas); this.repository.getAccountRepository().modifyAssetBalances(accountBalanceDeltas);
} }
@ -2327,17 +2208,7 @@ public class Block {
List<BlockRewardCandidate> rewardCandidates = new ArrayList<>(); List<BlockRewardCandidate> rewardCandidates = new ArrayList<>();
// All online accounts // All online accounts
final List<ExpandedAccount> expandedAccounts; final List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
if (this.getBlockData().getHeight() < BlockChain.getInstance().getFixBatchRewardHeight()) {
expandedAccounts = this.getExpandedAccounts().stream().collect(Collectors.toList());
}
else {
expandedAccounts
= this.getExpandedAccounts().stream()
.filter(expandedAccount -> expandedAccount.isMinterMember)
.collect(Collectors.toList());
}
/* /*
* Distribution rules: * Distribution rules:
@ -2380,6 +2251,7 @@ public class Block {
// Select the correct set of share bins based on block height // Select the correct set of share bins based on block height
List<AccountLevelShareBin> accountLevelShareBinsForBlock = (this.blockData.getHeight() >= BlockChain.getInstance().getSharesByLevelV2Height()) ? List<AccountLevelShareBin> accountLevelShareBinsForBlock = (this.blockData.getHeight() >= BlockChain.getInstance().getSharesByLevelV2Height()) ?
BlockChain.getInstance().getAccountLevelShareBinsV2() : BlockChain.getInstance().getAccountLevelShareBinsV1(); BlockChain.getInstance().getAccountLevelShareBinsV2() : BlockChain.getInstance().getAccountLevelShareBinsV1();
// Determine reward candidates based on account level // Determine reward candidates based on account level
// This needs a deep copy, so the shares can be modified when tiers aren't activated yet // This needs a deep copy, so the shares can be modified when tiers aren't activated yet
List<AccountLevelShareBin> accountLevelShareBins = new ArrayList<>(); List<AccountLevelShareBin> accountLevelShareBins = new ArrayList<>();
@ -2462,7 +2334,7 @@ public class Block {
final long qoraHoldersShare = BlockChain.getInstance().getQoraHoldersShareAtHeight(this.blockData.getHeight()); final long qoraHoldersShare = BlockChain.getInstance().getQoraHoldersShareAtHeight(this.blockData.getHeight());
// Perform account-level-based reward scaling if appropriate // Perform account-level-based reward scaling if appropriate
if (!haveFounders && this.blockData.getHeight() < BlockChain.getInstance().getAdminsReplaceFoundersHeight() ) { if (!haveFounders) {
// Recalculate distribution ratios based on candidates // Recalculate distribution ratios based on candidates
// Nothing shared? This shouldn't happen // Nothing shared? This shouldn't happen
@ -2498,103 +2370,18 @@ public class Block {
} }
// Add founders as reward candidate if appropriate // Add founders as reward candidate if appropriate
if (haveFounders && this.blockData.getHeight() < BlockChain.getInstance().getAdminsReplaceFoundersHeight()) { if (haveFounders) {
// Yes: add to reward candidates list // Yes: add to reward candidates list
BlockRewardDistributor founderDistributor = (distributionAmount, balanceChanges) -> distributeBlockRewardShare(distributionAmount, onlineFounderAccounts, balanceChanges); BlockRewardDistributor founderDistributor = (distributionAmount, balanceChanges) -> distributeBlockRewardShare(distributionAmount, onlineFounderAccounts, balanceChanges);
final long foundersShare = 1_00000000 - totalShares; final long foundersShare = 1_00000000 - totalShares;
BlockRewardCandidate rewardCandidate = new BlockRewardCandidate("Founders", foundersShare, founderDistributor); BlockRewardCandidate rewardCandidate = new BlockRewardCandidate("Founders", foundersShare, founderDistributor);
rewardCandidates.add(rewardCandidate); rewardCandidates.add(rewardCandidate);
LOGGER.info("logging foundersShare prior to reward modifications {}",foundersShare);
}
else if (this.blockData.getHeight() >= BlockChain.getInstance().getAdminsReplaceFoundersHeight()) {
try (final Repository repository = RepositoryManager.getRepository()) {
GroupRepository groupRepository = repository.getGroupRepository();
List<Integer> mintingGroupIds = Groups.getGroupIdsToMint(BlockChain.getInstance(), this.blockData.getHeight());
// all minter admins
List<String> minterAdmins = Groups.getAllAdmins(groupRepository, mintingGroupIds);
// all minter admins that are online
List<ExpandedAccount> onlineMinterAdminAccounts
= expandedAccounts.stream()
.filter(expandedAccount -> minterAdmins.contains(expandedAccount.getMintingAccount().getAddress()))
.collect(Collectors.toList());
long minterAdminShare;
if( onlineMinterAdminAccounts.isEmpty() ) {
minterAdminShare = 0;
}
else {
BlockRewardDistributor minterAdminDistributor
= (distributionAmount, balanceChanges)
->
distributeBlockRewardShare(distributionAmount, onlineMinterAdminAccounts, balanceChanges);
long adminShare = 1_00000000 - totalShares;
LOGGER.info("initial total Shares: {}", totalShares);
LOGGER.info("logging adminShare after hardfork, this is the primary reward that will be split {}", adminShare);
minterAdminShare = adminShare / 2;
BlockRewardCandidate minterAdminRewardCandidate
= new BlockRewardCandidate("Minter Admins", minterAdminShare, minterAdminDistributor);
rewardCandidates.add(minterAdminRewardCandidate);
totalShares += minterAdminShare;
}
LOGGER.info("MINTER ADMIN SHARE: {}",minterAdminShare);
// all dev admins
List<String> devAdminAddresses
= groupRepository.getGroupAdmins(1).stream()
.map(GroupAdminData::getAdmin)
.collect(Collectors.toList());
LOGGER.info("Removing NULL Account Address, Dev Admin Count = {}", devAdminAddresses.size());
devAdminAddresses.removeIf( address -> Group.NULL_OWNER_ADDRESS.equals(address) );
LOGGER.info("Removed NULL Account Address, Dev Admin Count = {}", devAdminAddresses.size());
BlockRewardDistributor devAdminDistributor
= (distributionAmount, balanceChanges) -> distributeToAccounts(distributionAmount, devAdminAddresses, balanceChanges);
long devAdminShare = 1_00000000 - totalShares;
LOGGER.info("DEV ADMIN SHARE: {}",devAdminShare);
BlockRewardCandidate devAdminRewardCandidate
= new BlockRewardCandidate("Dev Admins", devAdminShare,devAdminDistributor);
rewardCandidates.add(devAdminRewardCandidate);
}
} }
return rewardCandidates; return rewardCandidates;
} }
/**
* Distribute To Accounts
*
* Merges distribute shares to a map of distribution shares.
*
* @param distributionAmount the amount to distribute
* @param accountAddressess the addresses to distribute to
* @param balanceChanges the map of distribution shares, this gets appended to
*
* @return the total amount mapped to addresses for distribution
*/
public static long distributeToAccounts(long distributionAmount, List<String> accountAddressess, Map<String, Long> balanceChanges) {
if( accountAddressess.isEmpty() ) return 0;
long distibutionShare = distributionAmount / accountAddressess.size();
for(String accountAddress : accountAddressess ) {
balanceChanges.merge(accountAddress, distibutionShare, Long::sum);
}
return distibutionShare * accountAddressess.size();
}
private static long distributeBlockRewardShare(long distributionAmount, List<ExpandedAccount> accounts, Map<String, Long> balanceChanges) { private static long distributeBlockRewardShare(long distributionAmount, List<ExpandedAccount> accounts, Map<String, Long> balanceChanges) {
// Collate all expanded accounts by minting account // Collate all expanded accounts by minting account
Map<String, List<ExpandedAccount>> accountsByMinter = new HashMap<>(); Map<String, List<ExpandedAccount>> accountsByMinter = new HashMap<>();
@ -2754,11 +2541,9 @@ public class Block {
return; return;
int minterLevel = Account.getRewardShareEffectiveMintingLevel(this.repository, this.getMinter().getPublicKey()); int minterLevel = Account.getRewardShareEffectiveMintingLevel(this.repository, this.getMinter().getPublicKey());
String minterAddress = Account.getRewardShareMintingAddress(this.repository, this.getMinter().getPublicKey());
LOGGER.debug(String.format("======= BLOCK %d (%.8s) =======", this.getBlockData().getHeight(), Base58.encode(this.getSignature()))); LOGGER.debug(String.format("======= BLOCK %d (%.8s) =======", this.getBlockData().getHeight(), Base58.encode(this.getSignature())));
LOGGER.debug(String.format("Timestamp: %d", this.getBlockData().getTimestamp())); LOGGER.debug(String.format("Timestamp: %d", this.getBlockData().getTimestamp()));
LOGGER.debug(String.format("Minter address: %s", minterAddress));
LOGGER.debug(String.format("Minter level: %d", minterLevel)); LOGGER.debug(String.format("Minter level: %d", minterLevel));
LOGGER.debug(String.format("Online accounts: %d", this.getBlockData().getOnlineAccountsCount())); LOGGER.debug(String.format("Online accounts: %d", this.getBlockData().getOnlineAccountsCount()));
LOGGER.debug(String.format("AT count: %d", this.getBlockData().getATCount())); LOGGER.debug(String.format("AT count: %d", this.getBlockData().getATCount()));

View File

@ -1,101 +0,0 @@
package org.qortal.block;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.eclipse.persistence.jaxb.JAXBContextFactory;
import org.eclipse.persistence.jaxb.UnmarshallerProperties;
import org.qortal.data.account.AccountBalanceData;
import org.qortal.repository.DataException;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.UnmarshalException;
import javax.xml.bind.Unmarshaller;
import javax.xml.transform.stream.StreamSource;
import java.io.InputStream;
import java.util.List;
import java.util.stream.Collectors;
/**
* Block 1333492
* <p>
* As described in InvalidBalanceBlocks.java, legacy bugs caused a small drift in account balances.
* This block adjusts any remaining differences between a clean reindex/resync and a recent bootstrap.
* <p>
* The block height 1333492 isn't significant - it's simply the height of a recent bootstrap at the
* time of development, so that the account balances could be accessed and compared against the same
* block in a reindexed db.
* <p>
* As with InvalidBalanceBlocks, the discrepancies are insignificant, except for a single
* account which has a 3.03 QORT discrepancy. This was due to the account being the first recipient
* of a name sale and encountering an early bug in this area.
* <p>
* The total offset for this block is 3.02816514 QORT.
*/
public final class Block1333492 {
private static final Logger LOGGER = LogManager.getLogger(Block1333492.class);
private static final String ACCOUNT_DELTAS_SOURCE = "block-1333492-deltas.json";
private static final List<AccountBalanceData> accountDeltas = readAccountDeltas();
private Block1333492() {
/* Do not instantiate */
}
@SuppressWarnings("unchecked")
private static List<AccountBalanceData> readAccountDeltas() {
Unmarshaller unmarshaller;
try {
// Create JAXB context aware of classes we need to unmarshal
JAXBContext jc = JAXBContextFactory.createContext(new Class[] {
AccountBalanceData.class
}, null);
// Create unmarshaller
unmarshaller = jc.createUnmarshaller();
// Set the unmarshaller media type to JSON
unmarshaller.setProperty(UnmarshallerProperties.MEDIA_TYPE, "application/json");
// Tell unmarshaller that there's no JSON root element in the JSON input
unmarshaller.setProperty(UnmarshallerProperties.JSON_INCLUDE_ROOT, false);
} catch (JAXBException e) {
String message = "Failed to setup unmarshaller to read block 1333492 deltas";
LOGGER.error(message, e);
throw new RuntimeException(message, e);
}
ClassLoader classLoader = BlockChain.class.getClassLoader();
InputStream in = classLoader.getResourceAsStream(ACCOUNT_DELTAS_SOURCE);
StreamSource jsonSource = new StreamSource(in);
try {
// Attempt to unmarshal JSON stream to BlockChain config
return (List<AccountBalanceData>) unmarshaller.unmarshal(jsonSource, AccountBalanceData.class).getValue();
} catch (UnmarshalException e) {
String message = "Failed to parse block 1333492 deltas";
LOGGER.error(message, e);
throw new RuntimeException(message, e);
} catch (JAXBException e) {
String message = "Unexpected JAXB issue while processing block 1333492 deltas";
LOGGER.error(message, e);
throw new RuntimeException(message, e);
}
}
public static void processFix(Block block) throws DataException {
block.repository.getAccountRepository().modifyAssetBalances(accountDeltas);
}
public static void orphanFix(Block block) throws DataException {
// Create inverse deltas
List<AccountBalanceData> inverseDeltas = accountDeltas.stream()
.map(delta -> new AccountBalanceData(delta.getAddress(), delta.getAssetId(), 0 - delta.getBalance()))
.collect(Collectors.toList());
block.repository.getAccountRepository().modifyAssetBalances(inverseDeltas);
}
}

View File

@ -71,7 +71,6 @@ public class BlockChain {
transactionV6Timestamp, transactionV6Timestamp,
disableReferenceTimestamp, disableReferenceTimestamp,
increaseOnlineAccountsDifficultyTimestamp, increaseOnlineAccountsDifficultyTimestamp,
decreaseOnlineAccountsDifficultyTimestamp,
onlineAccountMinterLevelValidationHeight, onlineAccountMinterLevelValidationHeight,
selfSponsorshipAlgoV1Height, selfSponsorshipAlgoV1Height,
selfSponsorshipAlgoV2Height, selfSponsorshipAlgoV2Height,
@ -81,20 +80,7 @@ public class BlockChain {
arbitraryOptionalFeeTimestamp, arbitraryOptionalFeeTimestamp,
unconfirmableRewardSharesHeight, unconfirmableRewardSharesHeight,
disableTransferPrivsTimestamp, disableTransferPrivsTimestamp,
enableTransferPrivsTimestamp, enableTransferPrivsTimestamp
cancelSellNameValidationTimestamp,
disableRewardshareHeight,
enableRewardshareHeight,
onlyMintWithNameHeight,
removeOnlyMintWithNameHeight,
groupMemberCheckHeight,
fixBatchRewardHeight,
adminsReplaceFoundersHeight,
nullGroupMembershipHeight,
ignoreLevelForRewardShareHeight,
adminQueryFixHeight,
multipleNamesPerAccountHeight,
mintedBlocksAdjustmentRemovalHeight
} }
// Custom transaction fees // Custom transaction fees
@ -114,8 +100,7 @@ public class BlockChain {
/** Whether to use legacy, broken RIPEMD160 implementation when converting public keys to addresses. */ /** Whether to use legacy, broken RIPEMD160 implementation when converting public keys to addresses. */
private boolean useBrokenMD160ForAddresses = false; private boolean useBrokenMD160ForAddresses = false;
/** This should get ignored and overwritten in the oneNamePerAccount(int blockchainHeight) method, /** Whether only one registered name is allowed per account. */
* because it is based on block height, not based on the genesis block.*/
private boolean oneNamePerAccount = false; private boolean oneNamePerAccount = false;
/** Checkpoints */ /** Checkpoints */
@ -216,13 +201,6 @@ public class BlockChain {
private int maxRewardSharesPerFounderMintingAccount; private int maxRewardSharesPerFounderMintingAccount;
private int founderEffectiveMintingLevel; private int founderEffectiveMintingLevel;
public static class IdsForHeight {
public int height;
public List<Integer> ids;
}
private List<IdsForHeight> mintingGroupIds;
/** Minimum time to retain online account signatures (ms) for block validity checks. */ /** Minimum time to retain online account signatures (ms) for block validity checks. */
private long onlineAccountSignaturesMinLifetime; private long onlineAccountSignaturesMinLifetime;
@ -233,10 +211,6 @@ public class BlockChain {
* featureTriggers because unit tests need to set this value via Reflection. */ * featureTriggers because unit tests need to set this value via Reflection. */
private long onlineAccountsModulusV2Timestamp; private long onlineAccountsModulusV2Timestamp;
/** Feature trigger timestamp for ONLINE_ACCOUNTS_MODULUS time interval decrease. Can't use
* featureTriggers because unit tests need to set this value via Reflection. */
private long onlineAccountsModulusV3Timestamp;
/** Snapshot timestamp for self sponsorship algo V1 */ /** Snapshot timestamp for self sponsorship algo V1 */
private long selfSponsorshipAlgoV1SnapshotTimestamp; private long selfSponsorshipAlgoV1SnapshotTimestamp;
@ -423,9 +397,6 @@ public class BlockChain {
return this.onlineAccountsModulusV2Timestamp; return this.onlineAccountsModulusV2Timestamp;
} }
public long getOnlineAccountsModulusV3Timestamp() {
return this.onlineAccountsModulusV3Timestamp;
}
/* Block reward batching */ /* Block reward batching */
public long getBlockRewardBatchStartHeight() { public long getBlockRewardBatchStartHeight() {
@ -477,9 +448,8 @@ public class BlockChain {
return this.useBrokenMD160ForAddresses; return this.useBrokenMD160ForAddresses;
} }
public boolean oneNamePerAccount(int blockchainHeight) { public boolean oneNamePerAccount() {
// this is not set on a simple blockchain setting, it is based on a feature trigger height return this.oneNamePerAccount;
return blockchainHeight < this.getMultipleNamesPerAccountHeight();
} }
public List<Checkpoint> getCheckpoints() { public List<Checkpoint> getCheckpoints() {
@ -554,10 +524,6 @@ public class BlockChain {
return this.onlineAccountSignaturesMaxLifetime; return this.onlineAccountSignaturesMaxLifetime;
} }
public List<IdsForHeight> getMintingGroupIds() {
return mintingGroupIds;
}
public CiyamAtSettings getCiyamAtSettings() { public CiyamAtSettings getCiyamAtSettings() {
return this.ciyamAtSettings; return this.ciyamAtSettings;
} }
@ -604,10 +570,6 @@ public class BlockChain {
return this.featureTriggers.get(FeatureTrigger.increaseOnlineAccountsDifficultyTimestamp.name()).longValue(); return this.featureTriggers.get(FeatureTrigger.increaseOnlineAccountsDifficultyTimestamp.name()).longValue();
} }
public long getDecreaseOnlineAccountsDifficultyTimestamp() {
return this.featureTriggers.get(FeatureTrigger.decreaseOnlineAccountsDifficultyTimestamp.name()).longValue();
}
public int getSelfSponsorshipAlgoV1Height() { public int getSelfSponsorshipAlgoV1Height() {
return this.featureTriggers.get(FeatureTrigger.selfSponsorshipAlgoV1Height.name()).intValue(); return this.featureTriggers.get(FeatureTrigger.selfSponsorshipAlgoV1Height.name()).intValue();
} }
@ -648,58 +610,6 @@ public class BlockChain {
return this.featureTriggers.get(FeatureTrigger.enableTransferPrivsTimestamp.name()).longValue(); return this.featureTriggers.get(FeatureTrigger.enableTransferPrivsTimestamp.name()).longValue();
} }
public long getCancelSellNameValidationTimestamp() {
return this.featureTriggers.get(FeatureTrigger.cancelSellNameValidationTimestamp.name()).longValue();
}
public int getDisableRewardshareHeight() {
return this.featureTriggers.get(FeatureTrigger.disableRewardshareHeight.name()).intValue();
}
public int getEnableRewardshareHeight() {
return this.featureTriggers.get(FeatureTrigger.enableRewardshareHeight.name()).intValue();
}
public int getOnlyMintWithNameHeight() {
return this.featureTriggers.get(FeatureTrigger.onlyMintWithNameHeight.name()).intValue();
}
public int getRemoveOnlyMintWithNameHeight() {
return this.featureTriggers.get(FeatureTrigger.removeOnlyMintWithNameHeight.name()).intValue();
}
public int getGroupMemberCheckHeight() {
return this.featureTriggers.get(FeatureTrigger.groupMemberCheckHeight.name()).intValue();
}
public int getFixBatchRewardHeight() {
return this.featureTriggers.get(FeatureTrigger.fixBatchRewardHeight.name()).intValue();
}
public int getAdminsReplaceFoundersHeight() {
return this.featureTriggers.get(FeatureTrigger.adminsReplaceFoundersHeight.name()).intValue();
}
public int getNullGroupMembershipHeight() {
return this.featureTriggers.get(FeatureTrigger.nullGroupMembershipHeight.name()).intValue();
}
public int getIgnoreLevelForRewardShareHeight() {
return this.featureTriggers.get(FeatureTrigger.ignoreLevelForRewardShareHeight.name()).intValue();
}
public int getAdminQueryFixHeight() {
return this.featureTriggers.get(FeatureTrigger.adminQueryFixHeight.name()).intValue();
}
public int getMultipleNamesPerAccountHeight() {
return this.featureTriggers.get(FeatureTrigger.multipleNamesPerAccountHeight.name()).intValue();
}
public int getMintedBlocksAdjustmentRemovalHeight() {
return this.featureTriggers.get(FeatureTrigger.mintedBlocksAdjustmentRemovalHeight.name()).intValue();
}
// More complex getters for aspects that change by height or timestamp // More complex getters for aspects that change by height or timestamp
public long getRewardAtHeight(int ourHeight) { public long getRewardAtHeight(int ourHeight) {
@ -895,12 +805,10 @@ public class BlockChain {
boolean isLite = Settings.getInstance().isLite(); boolean isLite = Settings.getInstance().isLite();
boolean canBootstrap = Settings.getInstance().getBootstrap(); boolean canBootstrap = Settings.getInstance().getBootstrap();
boolean needsArchiveRebuild = false; boolean needsArchiveRebuild = false;
int checkHeight = 0;
BlockData chainTip; BlockData chainTip;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
chainTip = repository.getBlockRepository().getLastBlock(); chainTip = repository.getBlockRepository().getLastBlock();
checkHeight = repository.getBlockRepository().getBlockchainHeight();
// Ensure archive is (at least partially) intact, and force a bootstrap if it isn't // Ensure archive is (at least partially) intact, and force a bootstrap if it isn't
if (!isTopOnly && archiveEnabled && canBootstrap) { if (!isTopOnly && archiveEnabled && canBootstrap) {
@ -916,17 +824,6 @@ public class BlockChain {
} }
} }
if (!canBootstrap) {
if (checkHeight > 2) {
LOGGER.info("Retrieved block 2 from archive. Syncing from genesis block resumed!");
} else {
needsArchiveRebuild = (repository.getBlockArchiveRepository().fromHeight(2) == null);
if (needsArchiveRebuild) {
LOGGER.info("Couldn't retrieve block 2 from archive. Bootstrapping is disabled. Syncing from genesis block!");
}
}
}
// Validate checkpoints // Validate checkpoints
// Limited to topOnly nodes for now, in order to reduce risk, and to solve a real-world problem with divergent topOnly nodes // Limited to topOnly nodes for now, in order to reduce risk, and to solve a real-world problem with divergent topOnly nodes
// TODO: remove the isTopOnly conditional below once this feature has had more testing time // TODO: remove the isTopOnly conditional below once this feature has had more testing time
@ -959,12 +856,11 @@ public class BlockChain {
// Check first block is Genesis Block // Check first block is Genesis Block
if (!isGenesisBlockValid() || needsArchiveRebuild) { if (!isGenesisBlockValid() || needsArchiveRebuild) {
if (checkHeight < 3) { try {
try { rebuildBlockchain();
rebuildBlockchain();
} catch (InterruptedException e) { } catch (InterruptedException e) {
throw new DataException(String.format("Interrupted when trying to rebuild blockchain: %s", e.getMessage())); throw new DataException(String.format("Interrupted when trying to rebuild blockchain: %s", e.getMessage()));
}
} }
} }
@ -1105,4 +1001,5 @@ public class BlockChain {
blockchainLock.unlock(); blockchainLock.unlock();
} }
} }
} }

View File

@ -1,134 +0,0 @@
package org.qortal.block;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.eclipse.persistence.jaxb.JAXBContextFactory;
import org.eclipse.persistence.jaxb.UnmarshallerProperties;
import org.qortal.data.account.AccountBalanceData;
import org.qortal.repository.DataException;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.UnmarshalException;
import javax.xml.bind.Unmarshaller;
import javax.xml.transform.stream.StreamSource;
import java.io.InputStream;
import java.util.*;
import java.util.stream.Collectors;
/**
* Due to various bugs - which have been fixed - a small amount of balance drift occurred
* in the chainstate of running nodes and bootstraps, when compared with a clean sync from genesis.
* This resulted in a significant number of invalid transactions in the chain history due to
* subtle balance discrepancies. The sum of all discrepancies that resulted in an invalid
* transaction is 0.00198322 QORT, so despite the large quantity of transactions, they
* represent an insignificant amount when summed.
* <p>
* This class is responsible for retroactively fixing all the past transactions which
* are invalid due to the balance discrepancies.
*/
public final class InvalidBalanceBlocks {
private static final Logger LOGGER = LogManager.getLogger(InvalidBalanceBlocks.class);
private static final String ACCOUNT_DELTAS_SOURCE = "invalid-transaction-balance-deltas.json";
private static final List<AccountBalanceData> accountDeltas = readAccountDeltas();
private static final List<Integer> affectedHeights = getAffectedHeights();
private InvalidBalanceBlocks() {
/* Do not instantiate */
}
@SuppressWarnings("unchecked")
private static List<AccountBalanceData> readAccountDeltas() {
Unmarshaller unmarshaller;
try {
// Create JAXB context aware of classes we need to unmarshal
JAXBContext jc = JAXBContextFactory.createContext(new Class[] {
AccountBalanceData.class
}, null);
// Create unmarshaller
unmarshaller = jc.createUnmarshaller();
// Set the unmarshaller media type to JSON
unmarshaller.setProperty(UnmarshallerProperties.MEDIA_TYPE, "application/json");
// Tell unmarshaller that there's no JSON root element in the JSON input
unmarshaller.setProperty(UnmarshallerProperties.JSON_INCLUDE_ROOT, false);
} catch (JAXBException e) {
String message = "Failed to setup unmarshaller to read block 212937 deltas";
LOGGER.error(message, e);
throw new RuntimeException(message, e);
}
ClassLoader classLoader = BlockChain.class.getClassLoader();
InputStream in = classLoader.getResourceAsStream(ACCOUNT_DELTAS_SOURCE);
StreamSource jsonSource = new StreamSource(in);
try {
// Attempt to unmarshal JSON stream to BlockChain config
return (List<AccountBalanceData>) unmarshaller.unmarshal(jsonSource, AccountBalanceData.class).getValue();
} catch (UnmarshalException e) {
String message = "Failed to parse balance deltas";
LOGGER.error(message, e);
throw new RuntimeException(message, e);
} catch (JAXBException e) {
String message = "Unexpected JAXB issue while processing balance deltas";
LOGGER.error(message, e);
throw new RuntimeException(message, e);
}
}
private static List<Integer> getAffectedHeights() {
List<Integer> heights = new ArrayList<>();
for (AccountBalanceData accountBalanceData : accountDeltas) {
if (!heights.contains(accountBalanceData.getHeight())) {
heights.add(accountBalanceData.getHeight());
}
}
return heights;
}
private static List<AccountBalanceData> getAccountDeltasAtHeight(int height) {
return accountDeltas.stream().filter(a -> a.getHeight() == height).collect(Collectors.toList());
}
public static boolean isAffectedBlock(int height) {
return affectedHeights.contains(Integer.valueOf(height));
}
public static void processFix(Block block) throws DataException {
Integer blockHeight = block.getBlockData().getHeight();
List<AccountBalanceData> deltas = getAccountDeltasAtHeight(blockHeight);
if (deltas == null) {
throw new DataException(String.format("Unable to lookup invalid balance data for block height %d", blockHeight));
}
block.repository.getAccountRepository().modifyAssetBalances(deltas);
LOGGER.info("Applied balance patch for block {}", blockHeight);
}
public static void orphanFix(Block block) throws DataException {
Integer blockHeight = block.getBlockData().getHeight();
List<AccountBalanceData> deltas = getAccountDeltasAtHeight(blockHeight);
if (deltas == null) {
throw new DataException(String.format("Unable to lookup invalid balance data for block height %d", blockHeight));
}
// Create inverse delta(s)
for (AccountBalanceData accountBalanceData : deltas) {
AccountBalanceData inverseBalanceData = new AccountBalanceData(accountBalanceData.getAddress(), accountBalanceData.getAssetId(), -accountBalanceData.getBalance());
block.repository.getAccountRepository().modifyAssetBalances(List.of(inverseBalanceData));
}
LOGGER.info("Reverted balance patch for block {}", blockHeight);
}
}

View File

@ -1,47 +0,0 @@
package org.qortal.block;
import org.qortal.account.Account;
import org.qortal.api.resource.TransactionsResource;
import org.qortal.data.naming.NameData;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import java.util.Set;
import java.util.stream.Collectors;
/**
* Class PrimaryNamesBlock
*/
public class PrimaryNamesBlock {
/**
* Process Primary Names
*
* @param repository
* @throws DataException
*/
public static void processNames(Repository repository) throws DataException {
Set<String> addressesWithNames
= repository.getNameRepository().getAllNames().stream()
.map(NameData::getOwner).collect(Collectors.toSet());
// for each address with a name, set primary name to the address
for( String address : addressesWithNames ) {
Account account = new Account(repository, address);
account.resetPrimaryName(TransactionsResource.ConfirmationStatus.CONFIRMED);
}
}
/**
* Orphan the Primary Names Block
*
* @param repository
* @throws DataException
*/
public static void orphanNames(Repository repository) throws DataException {
repository.getNameRepository().clearPrimaryNames();
}
}

View File

@ -64,7 +64,6 @@ public class BlockMinter extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("BlockMinter"); Thread.currentThread().setName("BlockMinter");
Thread.currentThread().setPriority(MAX_PRIORITY);
if (Settings.getInstance().isTopOnly() || Settings.getInstance().isLite()) { if (Settings.getInstance().isTopOnly() || Settings.getInstance().isLite()) {
// Top only and lite nodes do not sign blocks // Top only and lite nodes do not sign blocks
@ -97,375 +96,364 @@ public class BlockMinter extends Thread {
final boolean isSingleNodeTestnet = Settings.getInstance().isSingleNodeTestnet(); final boolean isSingleNodeTestnet = Settings.getInstance().isSingleNodeTestnet();
// Flags for tracking change in whether minting is possible, try (final Repository repository = RepositoryManager.getRepository()) {
// so we can notify Controller, and further update SysTray, etc. // Going to need this a lot...
boolean isMintingPossible = false; BlockRepository blockRepository = repository.getBlockRepository();
boolean wasMintingPossible = isMintingPossible;
try { // Flags for tracking change in whether minting is possible,
// so we can notify Controller, and further update SysTray, etc.
boolean isMintingPossible = false;
boolean wasMintingPossible = isMintingPossible;
while (running) { while (running) {
// recreate repository for new loop iteration if (isMintingPossible != wasMintingPossible)
try (final Repository repository = RepositoryManager.getRepository()) { Controller.getInstance().onMintingPossibleChange(isMintingPossible);
// Going to need this a lot... wasMintingPossible = isMintingPossible;
BlockRepository blockRepository = repository.getBlockRepository();
if (isMintingPossible != wasMintingPossible) try {
Controller.getInstance().onMintingPossibleChange(isMintingPossible); // Free up any repository locks
repository.discardChanges();
wasMintingPossible = isMintingPossible; // Sleep for a while.
// It's faster on single node testnets, to allow lots of blocks to be minted quickly.
Thread.sleep(isSingleNodeTestnet ? 50 : 1000);
isMintingPossible = false;
final Long now = NTP.getTime();
if (now == null)
continue;
final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
if (minLatestBlockTimestamp == null)
continue;
List<MintingAccountData> mintingAccountsData = repository.getAccountRepository().getMintingAccounts();
// No minting accounts?
if (mintingAccountsData.isEmpty())
continue;
// Disregard minting accounts that are no longer valid, e.g. by transfer/loss of founder flag or account level
// Note that minting accounts are actually reward-shares in Qortal
Iterator<MintingAccountData> madi = mintingAccountsData.iterator();
while (madi.hasNext()) {
MintingAccountData mintingAccountData = madi.next();
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
if (rewardShareData == null) {
// Reward-share doesn't exist - probably cancelled but not yet removed from node's list of minting accounts
madi.remove();
continue;
}
Account mintingAccount = new Account(repository, rewardShareData.getMinter());
if (!mintingAccount.canMint()) {
// Minting-account component of reward-share can no longer mint - disregard
madi.remove();
continue;
}
// Optional (non-validated) prevention of block submissions below a defined level.
// This is an unvalidated version of Blockchain.minAccountLevelToMint
// and exists only to reduce block candidates by default.
int level = mintingAccount.getEffectiveMintingLevel();
if (level < BlockChain.getInstance().getMinAccountLevelForBlockSubmissions()) {
madi.remove();
}
}
// Needs a mutable copy of the unmodifiableList
List<Peer> peers = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
BlockData lastBlockData = blockRepository.getLastBlock();
// Disregard peers that have "misbehaved" recently
peers.removeIf(Controller.hasMisbehaved);
// Disregard peers that don't have a recent block, but only if we're not in recovery mode.
// In that mode, we want to allow minting on top of older blocks, to recover stalled networks.
if (!Synchronizer.getInstance().getRecoveryMode())
peers.removeIf(Controller.hasNoRecentBlock);
// Don't mint if we don't have enough up-to-date peers as where would the transactions/consensus come from?
if (peers.size() < Settings.getInstance().getMinBlockchainPeers())
continue;
// If we are stuck on an invalid block, we should allow an alternative to be minted
boolean recoverInvalidBlock = false;
if (Synchronizer.getInstance().timeInvalidBlockLastReceived != null) {
// We've had at least one invalid block
long timeSinceLastValidBlock = NTP.getTime() - Synchronizer.getInstance().timeValidBlockLastReceived;
long timeSinceLastInvalidBlock = NTP.getTime() - Synchronizer.getInstance().timeInvalidBlockLastReceived;
if (timeSinceLastValidBlock > INVALID_BLOCK_RECOVERY_TIMEOUT) {
if (timeSinceLastInvalidBlock < INVALID_BLOCK_RECOVERY_TIMEOUT) {
// Last valid block was more than 10 mins ago, but we've had an invalid block since then
// Assume that the chain has stalled because there is no alternative valid candidate
// Enter recovery mode to allow alternative, valid candidates to be minted
recoverInvalidBlock = true;
}
}
}
// If our latest block isn't recent then we need to synchronize instead of minting, unless we're in recovery mode.
if (!peers.isEmpty() && lastBlockData.getTimestamp() < minLatestBlockTimestamp)
if (!Synchronizer.getInstance().getRecoveryMode() && !recoverInvalidBlock)
continue;
// There are enough peers with a recent block and our latest block is recent
// so go ahead and mint a block if possible.
isMintingPossible = true;
// Check blockchain hasn't changed
if (previousBlockData == null || !Arrays.equals(previousBlockData.getSignature(), lastBlockData.getSignature())) {
previousBlockData = lastBlockData;
newBlocks.clear();
// Reduce log timeout
logTimeout = 10 * 1000L;
// Last low weight block is no longer valid
parentSignatureForLastLowWeightBlock = null;
}
// Discard accounts we have already built blocks with
mintingAccountsData.removeIf(mintingAccountData -> newBlocks.stream().anyMatch(newBlock -> Arrays.equals(newBlock.getBlockData().getMinterPublicKey(), mintingAccountData.getPublicKey())));
// Do we need to build any potential new blocks?
List<PrivateKeyAccount> newBlocksMintingAccounts = mintingAccountsData.stream().map(accountData -> new PrivateKeyAccount(repository, accountData.getPrivateKey())).collect(Collectors.toList());
// We might need to sit the next block out, if one of our minting accounts signed the previous one
// Skip this check for single node testnets, since they definitely need to mint every block
byte[] previousBlockMinter = previousBlockData.getMinterPublicKey();
boolean mintedLastBlock = mintingAccountsData.stream().anyMatch(mintingAccount -> Arrays.equals(mintingAccount.getPublicKey(), previousBlockMinter));
if (mintedLastBlock && !isSingleNodeTestnet) {
LOGGER.trace(String.format("One of our keys signed the last block, so we won't sign the next one"));
continue;
}
if (parentSignatureForLastLowWeightBlock != null) {
// The last iteration found a higher weight block in the network, so sleep for a while
// to allow is to sync the higher weight chain. We are sleeping here rather than when
// detected as we don't want to hold the blockchain lock open.
LOGGER.info("Sleeping for 10 seconds...");
Thread.sleep(10 * 1000L);
}
for (PrivateKeyAccount mintingAccount : newBlocksMintingAccounts) {
// First block does the AT heavy-lifting
if (newBlocks.isEmpty()) {
Block newBlock = Block.mint(repository, previousBlockData, mintingAccount);
if (newBlock == null) {
// For some reason we can't mint right now
moderatedLog(() -> LOGGER.info("Couldn't build a to-be-minted block"));
continue;
}
newBlocks.add(newBlock);
} else {
// The blocks for other minters require less effort...
Block newBlock = newBlocks.get(0).remint(mintingAccount);
if (newBlock == null) {
// For some reason we can't mint right now
moderatedLog(() -> LOGGER.error("Couldn't rebuild a to-be-minted block"));
continue;
}
newBlocks.add(newBlock);
}
}
// No potential block candidates?
if (newBlocks.isEmpty())
continue;
// Make sure we're the only thread modifying the blockchain
ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
if (!blockchainLock.tryLock(30, TimeUnit.SECONDS)) {
LOGGER.debug("Couldn't acquire blockchain lock even after waiting 30 seconds");
continue;
}
boolean newBlockMinted = false;
Block newBlock = null;
try { try {
// reset the repository, to the repository recreated for this loop iteration // Clear repository session state so we have latest view of data
for( Block newBlock : newBlocks ) newBlock.setRepository(repository);
// Free up any repository locks
repository.discardChanges(); repository.discardChanges();
// Sleep for a while. // Now that we have blockchain lock, do final check that chain hasn't changed
// It's faster on single node testnets, to allow lots of blocks to be minted quickly. BlockData latestBlockData = blockRepository.getLastBlock();
Thread.sleep(isSingleNodeTestnet ? 50 : 1000); if (!Arrays.equals(lastBlockData.getSignature(), latestBlockData.getSignature()))
isMintingPossible = false;
final Long now = NTP.getTime();
if (now == null)
continue; continue;
final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp(); List<Block> goodBlocks = new ArrayList<>();
if (minLatestBlockTimestamp == null) boolean wasInvalidBlockDiscarded = false;
continue; Iterator<Block> newBlocksIterator = newBlocks.iterator();
List<MintingAccountData> mintingAccountsData = repository.getAccountRepository().getMintingAccounts(); while (newBlocksIterator.hasNext()) {
// No minting accounts? Block testBlock = newBlocksIterator.next();
if (mintingAccountsData.isEmpty())
continue;
// Disregard minting accounts that are no longer valid, e.g. by transfer/loss of founder flag or account level // Is new block's timestamp valid yet?
// Note that minting accounts are actually reward-shares in Qortal // We do a separate check as some timestamp checks are skipped for testchains
Iterator<MintingAccountData> madi = mintingAccountsData.iterator(); if (testBlock.isTimestampValid() != ValidationResult.OK)
while (madi.hasNext()) {
MintingAccountData mintingAccountData = madi.next();
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
if (rewardShareData == null) {
// Reward-share doesn't exist - probably cancelled but not yet removed from node's list of minting accounts
madi.remove();
continue;
}
Account mintingAccount = new Account(repository, rewardShareData.getMinter());
if (!mintingAccount.canMint(true)) {
// Minting-account component of reward-share can no longer mint - disregard
madi.remove();
continue;
}
// Optional (non-validated) prevention of block submissions below a defined level.
// This is an unvalidated version of Blockchain.minAccountLevelToMint
// and exists only to reduce block candidates by default.
int level = mintingAccount.getEffectiveMintingLevel();
if (level < BlockChain.getInstance().getMinAccountLevelForBlockSubmissions()) {
madi.remove();
}
}
// Needs a mutable copy of the unmodifiableList
List<Peer> peers = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
BlockData lastBlockData = blockRepository.getLastBlock();
// Disregard peers that have "misbehaved" recently
peers.removeIf(Controller.hasMisbehaved);
// Disregard peers that don't have a recent block, but only if we're not in recovery mode.
// In that mode, we want to allow minting on top of older blocks, to recover stalled networks.
if (!Synchronizer.getInstance().getRecoveryMode())
peers.removeIf(Controller.hasNoRecentBlock);
// Don't mint if we don't have enough up-to-date peers as where would the transactions/consensus come from?
if (peers.size() < Settings.getInstance().getMinBlockchainPeers())
continue;
// If we are stuck on an invalid block, we should allow an alternative to be minted
boolean recoverInvalidBlock = false;
if (Synchronizer.getInstance().timeInvalidBlockLastReceived != null) {
// We've had at least one invalid block
long timeSinceLastValidBlock = NTP.getTime() - Synchronizer.getInstance().timeValidBlockLastReceived;
long timeSinceLastInvalidBlock = NTP.getTime() - Synchronizer.getInstance().timeInvalidBlockLastReceived;
if (timeSinceLastValidBlock > INVALID_BLOCK_RECOVERY_TIMEOUT) {
if (timeSinceLastInvalidBlock < INVALID_BLOCK_RECOVERY_TIMEOUT) {
// Last valid block was more than 10 mins ago, but we've had an invalid block since then
// Assume that the chain has stalled because there is no alternative valid candidate
// Enter recovery mode to allow alternative, valid candidates to be minted
recoverInvalidBlock = true;
}
}
}
// If our latest block isn't recent then we need to synchronize instead of minting, unless we're in recovery mode.
if (!peers.isEmpty() && lastBlockData.getTimestamp() < minLatestBlockTimestamp)
if (!Synchronizer.getInstance().getRecoveryMode() && !recoverInvalidBlock)
continue; continue;
// There are enough peers with a recent block and our latest block is recent testBlock.preProcess();
// so go ahead and mint a block if possible.
isMintingPossible = true;
// Check blockchain hasn't changed // Is new block valid yet? (Before adding unconfirmed transactions)
if (previousBlockData == null || !Arrays.equals(previousBlockData.getSignature(), lastBlockData.getSignature())) { ValidationResult result = testBlock.isValid();
previousBlockData = lastBlockData; if (result != ValidationResult.OK) {
newBlocks.clear(); moderatedLog(() -> LOGGER.error(String.format("To-be-minted block invalid '%s' before adding transactions?", result.name())));
// Reduce log timeout newBlocksIterator.remove();
logTimeout = 10 * 1000L; wasInvalidBlockDiscarded = true;
/*
* Bail out fast so that we loop around from the top again.
* This gives BlockMinter the possibility to remint this candidate block using another block from newBlocks,
* via the Blocks.remint() method, which avoids having to re-process Block ATs all over again.
* Particularly useful if some aspect of Blocks changes due a timestamp-based feature-trigger (see BlockChain class).
*/
break;
}
// Last low weight block is no longer valid goodBlocks.add(testBlock);
parentSignatureForLastLowWeightBlock = null;
} }
// Discard accounts we have already built blocks with if (wasInvalidBlockDiscarded || goodBlocks.isEmpty())
mintingAccountsData.removeIf(mintingAccountData -> newBlocks.stream().anyMatch(newBlock -> Arrays.equals(newBlock.getBlockData().getMinterPublicKey(), mintingAccountData.getPublicKey())));
// Do we need to build any potential new blocks?
List<PrivateKeyAccount> newBlocksMintingAccounts = mintingAccountsData.stream().map(accountData -> new PrivateKeyAccount(repository, accountData.getPrivateKey())).collect(Collectors.toList());
// We might need to sit the next block out, if one of our minting accounts signed the previous one
// Skip this check for single node testnets, since they definitely need to mint every block
byte[] previousBlockMinter = previousBlockData.getMinterPublicKey();
boolean mintedLastBlock = mintingAccountsData.stream().anyMatch(mintingAccount -> Arrays.equals(mintingAccount.getPublicKey(), previousBlockMinter));
if (mintedLastBlock && !isSingleNodeTestnet) {
LOGGER.trace(String.format("One of our keys signed the last block, so we won't sign the next one"));
continue; continue;
}
if (parentSignatureForLastLowWeightBlock != null) { // Pick best block
// The last iteration found a higher weight block in the network, so sleep for a while final int parentHeight = previousBlockData.getHeight();
// to allow is to sync the higher weight chain. We are sleeping here rather than when final byte[] parentBlockSignature = previousBlockData.getSignature();
// detected as we don't want to hold the blockchain lock open.
LOGGER.info("Sleeping for 10 seconds...");
Thread.sleep(10 * 1000L);
}
for (PrivateKeyAccount mintingAccount : newBlocksMintingAccounts) { BigInteger bestWeight = null;
// First block does the AT heavy-lifting
if (newBlocks.isEmpty()) {
Block newBlock = Block.mint(repository, previousBlockData, mintingAccount);
if (newBlock == null) {
// For some reason we can't mint right now
moderatedLog(() -> LOGGER.info("Couldn't build a to-be-minted block"));
continue;
}
newBlocks.add(newBlock); for (int bi = 0; bi < goodBlocks.size(); ++bi) {
} else { BlockData blockData = goodBlocks.get(bi).getBlockData();
// The blocks for other minters require less effort...
Block newBlock = newBlocks.get(0).remint(mintingAccount);
if (newBlock == null) {
// For some reason we can't mint right now
moderatedLog(() -> LOGGER.error("Couldn't rebuild a to-be-minted block"));
continue;
}
newBlocks.add(newBlock); BlockSummaryData blockSummaryData = new BlockSummaryData(blockData);
int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey());
blockSummaryData.setMinterLevel(minterLevel);
BigInteger blockWeight = Block.calcBlockWeight(parentHeight, parentBlockSignature, blockSummaryData);
if (bestWeight == null || blockWeight.compareTo(bestWeight) < 0) {
newBlock = goodBlocks.get(bi);
bestWeight = blockWeight;
} }
} }
// No potential block candidates?
if (newBlocks.isEmpty())
continue;
// Make sure we're the only thread modifying the blockchain
ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
if (!blockchainLock.tryLock(30, TimeUnit.SECONDS)) {
LOGGER.debug("Couldn't acquire blockchain lock even after waiting 30 seconds");
continue;
}
boolean newBlockMinted = false;
Block newBlock = null;
try { try {
// Clear repository session state so we have latest view of data if (this.higherWeightChainExists(repository, bestWeight)) {
repository.discardChanges();
// Now that we have blockchain lock, do final check that chain hasn't changed // Check if the base block has updated since the last time we were here
BlockData latestBlockData = blockRepository.getLastBlock(); if (parentSignatureForLastLowWeightBlock == null || timeOfLastLowWeightBlock == null ||
if (!Arrays.equals(lastBlockData.getSignature(), latestBlockData.getSignature())) !Arrays.equals(parentSignatureForLastLowWeightBlock, previousBlockData.getSignature())) {
continue; // We've switched to a different chain, so reset the timer
timeOfLastLowWeightBlock = NTP.getTime();
}
parentSignatureForLastLowWeightBlock = previousBlockData.getSignature();
List<Block> goodBlocks = new ArrayList<>(); // If less than 30 seconds has passed since first detection the higher weight chain,
boolean wasInvalidBlockDiscarded = false; // we should skip our block submission to give us the opportunity to sync to the better chain
Iterator<Block> newBlocksIterator = newBlocks.iterator(); if (NTP.getTime() - timeOfLastLowWeightBlock < 30 * 1000L) {
LOGGER.info("Higher weight chain found in peers, so not signing a block this round");
while (newBlocksIterator.hasNext()) { LOGGER.info("Time since detected: {}", NTP.getTime() - timeOfLastLowWeightBlock);
Block testBlock = newBlocksIterator.next();
// Is new block's timestamp valid yet?
// We do a separate check as some timestamp checks are skipped for testchains
if (testBlock.isTimestampValid() != ValidationResult.OK)
continue; continue;
testBlock.preProcess();
// Is new block valid yet? (Before adding unconfirmed transactions)
ValidationResult result = testBlock.isValid();
if (result != ValidationResult.OK) {
moderatedLog(() -> LOGGER.error(String.format("To-be-minted block invalid '%s' before adding transactions?", result.name())));
newBlocksIterator.remove();
wasInvalidBlockDiscarded = true;
/*
* Bail out fast so that we loop around from the top again.
* This gives BlockMinter the possibility to remint this candidate block using another block from newBlocks,
* via the Blocks.remint() method, which avoids having to re-process Block ATs all over again.
* Particularly useful if some aspect of Blocks changes due a timestamp-based feature-trigger (see BlockChain class).
*/
break;
}
goodBlocks.add(testBlock);
}
if (wasInvalidBlockDiscarded || goodBlocks.isEmpty())
continue;
// Pick best block
final int parentHeight = previousBlockData.getHeight();
final byte[] parentBlockSignature = previousBlockData.getSignature();
BigInteger bestWeight = null;
for (int bi = 0; bi < goodBlocks.size(); ++bi) {
BlockData blockData = goodBlocks.get(bi).getBlockData();
BlockSummaryData blockSummaryData = new BlockSummaryData(blockData);
int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey());
blockSummaryData.setMinterLevel(minterLevel);
BigInteger blockWeight = Block.calcBlockWeight(parentHeight, parentBlockSignature, blockSummaryData);
if (bestWeight == null || blockWeight.compareTo(bestWeight) < 0) {
newBlock = goodBlocks.get(bi);
bestWeight = blockWeight;
}
}
try {
if (this.higherWeightChainExists(repository, bestWeight)) {
// Check if the base block has updated since the last time we were here
if (parentSignatureForLastLowWeightBlock == null || timeOfLastLowWeightBlock == null ||
!Arrays.equals(parentSignatureForLastLowWeightBlock, previousBlockData.getSignature())) {
// We've switched to a different chain, so reset the timer
timeOfLastLowWeightBlock = NTP.getTime();
}
parentSignatureForLastLowWeightBlock = previousBlockData.getSignature();
// If less than 30 seconds has passed since first detection the higher weight chain,
// we should skip our block submission to give us the opportunity to sync to the better chain
if (NTP.getTime() - timeOfLastLowWeightBlock < 30 * 1000L) {
LOGGER.info("Higher weight chain found in peers, so not signing a block this round");
LOGGER.info("Time since detected: {}", NTP.getTime() - timeOfLastLowWeightBlock);
continue;
} else {
// More than 30 seconds have passed, so we should submit our block candidate anyway.
LOGGER.info("More than 30 seconds passed, so proceeding to submit block candidate...");
}
} else { } else {
LOGGER.debug("No higher weight chain found in peers"); // More than 30 seconds have passed, so we should submit our block candidate anyway.
LOGGER.info("More than 30 seconds passed, so proceeding to submit block candidate...");
} }
} catch (DataException e) { } else {
LOGGER.debug("Unable to check for a higher weight chain. Proceeding anyway..."); LOGGER.debug("No higher weight chain found in peers");
} }
} catch (DataException e) {
// Discard any uncommitted changes as a result of the higher weight chain detection LOGGER.debug("Unable to check for a higher weight chain. Proceeding anyway...");
repository.discardChanges();
// Clear variables that track low weight blocks
parentSignatureForLastLowWeightBlock = null;
timeOfLastLowWeightBlock = null;
Long unconfirmedStartTime = NTP.getTime();
// Add unconfirmed transactions
addUnconfirmedTransactions(repository, newBlock);
LOGGER.info(String.format("Adding %d unconfirmed transactions took %d ms", newBlock.getTransactions().size(), (NTP.getTime() - unconfirmedStartTime)));
// Sign to create block's signature
newBlock.sign();
// Is newBlock still valid?
ValidationResult validationResult = newBlock.isValid();
if (validationResult != ValidationResult.OK) {
// No longer valid? Report and discard
LOGGER.error(String.format("To-be-minted block now invalid '%s' after adding unconfirmed transactions?", validationResult.name()));
// Rebuild block candidates, just to be sure
newBlocks.clear();
continue;
}
// Add to blockchain - something else will notice and broadcast new block to network
try {
newBlock.process();
repository.saveChanges();
LOGGER.info(String.format("Minted new block: %d", newBlock.getBlockData().getHeight()));
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(newBlock.getBlockData().getMinterPublicKey());
if (rewardShareData != null) {
LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s on behalf of %s",
newBlock.getBlockData().getHeight(),
Base58.encode(newBlock.getBlockData().getSignature()),
Base58.encode(newBlock.getParent().getSignature()),
rewardShareData.getMinter(),
rewardShareData.getRecipient()));
} else {
LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s",
newBlock.getBlockData().getHeight(),
Base58.encode(newBlock.getBlockData().getSignature()),
Base58.encode(newBlock.getParent().getSignature()),
newBlock.getMinter().getAddress()));
}
// Notify network after we're released blockchain lock
newBlockMinted = true;
// Notify Controller
repository.discardChanges(); // clear transaction status to prevent deadlocks
Controller.getInstance().onNewBlock(newBlock.getBlockData());
} catch (DataException e) {
// Unable to process block - report and discard
LOGGER.error("Unable to process newly minted block?", e);
newBlocks.clear();
} catch (ArithmeticException e) {
// Unable to process block - report and discard
LOGGER.error("Unable to process newly minted block?", e);
newBlocks.clear();
}
} finally {
blockchainLock.unlock();
} }
if (newBlockMinted) { // Discard any uncommitted changes as a result of the higher weight chain detection
// Broadcast our new chain to network repository.discardChanges();
Network.getInstance().broadcastOurChain();
// Clear variables that track low weight blocks
parentSignatureForLastLowWeightBlock = null;
timeOfLastLowWeightBlock = null;
Long unconfirmedStartTime = NTP.getTime();
// Add unconfirmed transactions
addUnconfirmedTransactions(repository, newBlock);
LOGGER.info(String.format("Adding %d unconfirmed transactions took %d ms", newBlock.getTransactions().size(), (NTP.getTime()-unconfirmedStartTime)));
// Sign to create block's signature
newBlock.sign();
// Is newBlock still valid?
ValidationResult validationResult = newBlock.isValid();
if (validationResult != ValidationResult.OK) {
// No longer valid? Report and discard
LOGGER.error(String.format("To-be-minted block now invalid '%s' after adding unconfirmed transactions?", validationResult.name()));
// Rebuild block candidates, just to be sure
newBlocks.clear();
continue;
} }
} catch (InterruptedException e) { // Add to blockchain - something else will notice and broadcast new block to network
// We've been interrupted - time to exit try {
return; newBlock.process();
repository.saveChanges();
LOGGER.info(String.format("Minted new block: %d", newBlock.getBlockData().getHeight()));
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(newBlock.getBlockData().getMinterPublicKey());
if (rewardShareData != null) {
LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s on behalf of %s",
newBlock.getBlockData().getHeight(),
Base58.encode(newBlock.getBlockData().getSignature()),
Base58.encode(newBlock.getParent().getSignature()),
rewardShareData.getMinter(),
rewardShareData.getRecipient()));
} else {
LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s",
newBlock.getBlockData().getHeight(),
Base58.encode(newBlock.getBlockData().getSignature()),
Base58.encode(newBlock.getParent().getSignature()),
newBlock.getMinter().getAddress()));
}
// Notify network after we're released blockchain lock
newBlockMinted = true;
// Notify Controller
repository.discardChanges(); // clear transaction status to prevent deadlocks
Controller.getInstance().onNewBlock(newBlock.getBlockData());
} catch (DataException e) {
// Unable to process block - report and discard
LOGGER.error("Unable to process newly minted block?", e);
newBlocks.clear();
} catch (ArithmeticException e) {
// Unable to process block - report and discard
LOGGER.error("Unable to process newly minted block?", e);
newBlocks.clear();
}
} finally {
blockchainLock.unlock();
} }
} catch (DataException e) {
LOGGER.warn("Repository issue while running block minter - NO LONGER MINTING", e); if (newBlockMinted) {
} catch (Exception e) { // Broadcast our new chain to network
LOGGER.error(e.getMessage(), e); Network.getInstance().broadcastOurChain();
}
} catch (InterruptedException e) {
// We've been interrupted - time to exit
return;
} }
} }
} catch (Exception e) { } catch (DataException e) {
LOGGER.error(e.getMessage(), e); LOGGER.warn("Repository issue while running block minter - NO LONGER MINTING", e);
} }
} }

View File

@ -13,8 +13,6 @@ import org.qortal.block.Block;
import org.qortal.block.BlockChain; import org.qortal.block.BlockChain;
import org.qortal.block.BlockChain.BlockTimingByHeight; import org.qortal.block.BlockChain.BlockTimingByHeight;
import org.qortal.controller.arbitrary.*; import org.qortal.controller.arbitrary.*;
import org.qortal.controller.hsqldb.HSQLDBBalanceRecorder;
import org.qortal.controller.hsqldb.HSQLDBDataCacheManager;
import org.qortal.controller.repository.NamesDatabaseIntegrityCheck; import org.qortal.controller.repository.NamesDatabaseIntegrityCheck;
import org.qortal.controller.repository.PruneManager; import org.qortal.controller.repository.PruneManager;
import org.qortal.controller.tradebot.TradeBot; import org.qortal.controller.tradebot.TradeBot;
@ -34,7 +32,6 @@ import org.qortal.gui.Gui;
import org.qortal.gui.SysTray; import org.qortal.gui.SysTray;
import org.qortal.network.Network; import org.qortal.network.Network;
import org.qortal.network.Peer; import org.qortal.network.Peer;
import org.qortal.network.PeerAddress;
import org.qortal.network.message.*; import org.qortal.network.message.*;
import org.qortal.repository.*; import org.qortal.repository.*;
import org.qortal.repository.hsqldb.HSQLDBRepositoryFactory; import org.qortal.repository.hsqldb.HSQLDBRepositoryFactory;
@ -46,18 +43,13 @@ import org.qortal.utils.*;
import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType; import javax.xml.bind.annotation.XmlAccessorType;
import java.awt.TrayIcon.MessageType; import java.awt.TrayIcon.MessageType;
import java.io.File; import java.io.File;
import java.io.FileNotFoundException; import java.io.FileNotFoundException;
import java.io.IOException; import java.io.IOException;
import java.io.InputStream; import java.io.InputStream;
import java.net.InetSocketAddress;
import java.net.UnknownHostException;
import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.security.SecureRandom;
import java.security.Security; import java.security.Security;
import java.time.LocalDateTime; import java.time.LocalDateTime;
import java.time.ZoneOffset; import java.time.ZoneOffset;
@ -72,7 +64,6 @@ import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Function; import java.util.function.Function;
import java.util.function.Predicate; import java.util.function.Predicate;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import java.util.stream.Stream;
public class Controller extends Thread { public class Controller extends Thread {
@ -104,7 +95,7 @@ public class Controller extends Thread {
private final long buildTimestamp; // seconds private final long buildTimestamp; // seconds
private final String[] savedArgs; private final String[] savedArgs;
private ExecutorService callbackExecutor = Executors.newFixedThreadPool(4); private ExecutorService callbackExecutor = Executors.newFixedThreadPool(3);
private volatile boolean notifyGroupMembershipChange = false; private volatile boolean notifyGroupMembershipChange = false;
/** Latest blocks on our chain. Note: tail/last is the latest block. */ /** Latest blocks on our chain. Note: tail/last is the latest block. */
@ -397,9 +388,6 @@ public class Controller extends Thread {
Controller.newInstance(args); Controller.newInstance(args);
cleanChunkUploadTempDir(); // cleanup leftover chunks from streaming to disk
LOGGER.info("Starting NTP"); LOGGER.info("Starting NTP");
Long ntpOffset = Settings.getInstance().getTestNtpOffset(); Long ntpOffset = Settings.getInstance().getTestNtpOffset();
if (ntpOffset != null) if (ntpOffset != null)
@ -409,44 +397,14 @@ public class Controller extends Thread {
LOGGER.info("Starting repository"); LOGGER.info("Starting repository");
try { try {
HSQLDBRepositoryFactory repositoryFactory = new HSQLDBRepositoryFactory(getRepositoryUrl()); RepositoryFactory repositoryFactory = new HSQLDBRepositoryFactory(getRepositoryUrl());
RepositoryManager.setRepositoryFactory(repositoryFactory); RepositoryManager.setRepositoryFactory(repositoryFactory);
RepositoryManager.setRequestedCheckpoint(Boolean.TRUE); RepositoryManager.setRequestedCheckpoint(Boolean.TRUE);
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
// RepositoryManager.rebuildTransactionSequences(repository); RepositoryManager.rebuildTransactionSequences(repository);
ArbitraryDataCacheManager.getInstance().buildArbitraryResourcesCache(repository, false); ArbitraryDataCacheManager.getInstance().buildArbitraryResourcesCache(repository, false);
} }
if( Settings.getInstance().isDbCacheEnabled() ) {
LOGGER.info("Db Cache Starting ...");
HSQLDBDataCacheManager hsqldbDataCacheManager = new HSQLDBDataCacheManager();
hsqldbDataCacheManager.start();
}
else {
LOGGER.info("Db Cache Disabled");
}
LOGGER.info("Arbitrary Indexing Starting ...");
ArbitraryIndexUtils.startCaching(
Settings.getInstance().getArbitraryIndexingPriority(),
Settings.getInstance().getArbitraryIndexingFrequency()
);
if( Settings.getInstance().isBalanceRecorderEnabled() ) {
Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
if( recorder.isPresent() ) {
LOGGER.info("Balance Recorder Starting ...");
recorder.get().start();
}
else {
LOGGER.info("Balance Recorder won't start.");
}
}
else {
LOGGER.info("Balance Recorder Disabled");
}
} catch (DataException e) { } catch (DataException e) {
// If exception has no cause or message then repository is in use by some other process. // If exception has no cause or message then repository is in use by some other process.
if (e.getCause() == null && e.getMessage() == null) { if (e.getCause() == null && e.getMessage() == null) {
@ -527,6 +485,7 @@ public class Controller extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Shutdown hook"); Thread.currentThread().setName("Shutdown hook");
Controller.getInstance().shutdown(); Controller.getInstance().shutdown();
} }
}); });
@ -551,25 +510,9 @@ public class Controller extends Thread {
ArbitraryDataStorageManager.getInstance().start(); ArbitraryDataStorageManager.getInstance().start();
ArbitraryDataRenderManager.getInstance().start(); ArbitraryDataRenderManager.getInstance().start();
// start rebuild arbitrary resource cache timer task
if( Settings.getInstance().isRebuildArbitraryResourceCacheTaskEnabled() ) {
new Timer().schedule(
new RebuildArbitraryResourceCacheTask(),
Settings.getInstance().getRebuildArbitraryResourceCacheTaskDelay() * RebuildArbitraryResourceCacheTask.MILLIS_IN_MINUTE,
Settings.getInstance().getRebuildArbitraryResourceCacheTaskPeriod() * RebuildArbitraryResourceCacheTask.MILLIS_IN_HOUR
);
}
LOGGER.info("Starting online accounts manager"); LOGGER.info("Starting online accounts manager");
OnlineAccountsManager.getInstance().start(); OnlineAccountsManager.getInstance().start();
LOGGER.info("Starting foreign fees manager");
ForeignFeesManager.getInstance().start();
LOGGER.info("Starting follower");
Follower.getInstance().start();
LOGGER.info("Starting transaction importer"); LOGGER.info("Starting transaction importer");
TransactionImporter.getInstance().start(); TransactionImporter.getInstance().start();
@ -622,33 +565,10 @@ public class Controller extends Thread {
// If GUI is enabled, we're no longer starting up but actually running now // If GUI is enabled, we're no longer starting up but actually running now
Gui.getInstance().notifyRunning(); Gui.getInstance().notifyRunning();
if (Settings.getInstance().isAutoRestartEnabled()) {
// Check every 10 minutes if we have enough connected peers
Timer checkConnectedPeers = new Timer();
checkConnectedPeers.schedule(new TimerTask() {
@Override
public void run() {
// Get the connected peers
int myConnectedPeers = Network.getInstance().getImmutableHandshakedPeers().size();
LOGGER.debug("Node have {} connected peers", myConnectedPeers);
if (myConnectedPeers == 0) {
// Restart node if we have 0 peers
LOGGER.info("Node have no connected peers, restarting node");
try {
RestartNode.attemptToRestart();
} catch (Exception e) {
LOGGER.error("Unable to restart the node", e);
}
}
}
}, 10*60*1000, 10*60*1000);
}
// Check every 10 minutes to see if the block minter is running // Check every 10 minutes to see if the block minter is running
Timer checkBlockMinter = new Timer(); Timer timer = new Timer();
checkBlockMinter.schedule(new TimerTask() { timer.schedule(new TimerTask() {
@Override @Override
public void run() { public void run() {
if (blockMinter.isAlive()) { if (blockMinter.isAlive()) {
@ -672,71 +592,6 @@ public class Controller extends Thread {
} }
} }
}, 10*60*1000, 10*60*1000); }, 10*60*1000, 10*60*1000);
// Check if we need sync from genesis and start syncing
Timer syncFromGenesis = new Timer();
syncFromGenesis.schedule(new TimerTask() {
@Override
public void run() {
LOGGER.debug("Start sync from genesis check.");
boolean canBootstrap = Settings.getInstance().getBootstrap();
boolean needsArchiveRebuild = false;
int checkHeight = 0;
try (final Repository repository = RepositoryManager.getRepository()){
needsArchiveRebuild = (repository.getBlockArchiveRepository().fromHeight(2) == null);
checkHeight = repository.getBlockRepository().getBlockchainHeight();
} catch (DataException e) {
throw new RuntimeException(e);
}
if (canBootstrap || !needsArchiveRebuild || checkHeight > 3) {
LOGGER.debug("Bootstrapping is enabled or we have more than 2 blocks, cancel sync from genesis check.");
syncFromGenesis.cancel();
return;
}
if (needsArchiveRebuild && !canBootstrap) {
LOGGER.info("Start syncing from genesis!");
List<Peer> seeds = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
// Check if have a qualified peer to sync
if (seeds.isEmpty()) {
LOGGER.info("No connected peers, will try again later.");
return;
}
int index = new SecureRandom().nextInt(seeds.size());
String syncNode = String.valueOf(seeds.get(index));
PeerAddress peerAddress = PeerAddress.fromString(syncNode);
InetSocketAddress resolvedAddress = null;
try {
resolvedAddress = peerAddress.toSocketAddress();
} catch (UnknownHostException e) {
throw new RuntimeException(e);
}
InetSocketAddress finalResolvedAddress = resolvedAddress;
Peer targetPeer = seeds.stream().filter(peer -> peer.getResolvedAddress().equals(finalResolvedAddress)).findFirst().orElse(null);
Synchronizer.SynchronizationResult syncResult;
try {
do {
try {
syncResult = Synchronizer.getInstance().actuallySynchronize(targetPeer, true);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
while (syncResult == Synchronizer.SynchronizationResult.OK);
} finally {
// We are syncing now, so can cancel the check
syncFromGenesis.cancel();
}
}
}
}, 3*60*1000, 3*60*1000);
} }
/** Called by AdvancedInstaller's launch EXE in single-instance mode, when an instance is already running. */ /** Called by AdvancedInstaller's launch EXE in single-instance mode, when an instance is already running. */
@ -1140,9 +995,6 @@ public class Controller extends Thread {
LOGGER.info("Shutting down online accounts manager"); LOGGER.info("Shutting down online accounts manager");
OnlineAccountsManager.getInstance().shutdown(); OnlineAccountsManager.getInstance().shutdown();
LOGGER.info("Shutting down foreign fees manager");
ForeignFeesManager.getInstance().shutdown();
LOGGER.info("Shutting down transaction importer"); LOGGER.info("Shutting down transaction importer");
TransactionImporter.getInstance().shutdown(); TransactionImporter.getInstance().shutdown();
@ -1487,14 +1339,6 @@ public class Controller extends Thread {
OnlineAccountsManager.getInstance().onNetworkOnlineAccountsV3Message(peer, message); OnlineAccountsManager.getInstance().onNetworkOnlineAccountsV3Message(peer, message);
break; break;
case GET_FOREIGN_FEES:
ForeignFeesManager.getInstance().onNetworkGetForeignFeesMessage(peer, message);
break;
case FOREIGN_FEES:
ForeignFeesManager.getInstance().onNetworkForeignFeesMessage(peer, message);
break;
case GET_ARBITRARY_DATA: case GET_ARBITRARY_DATA:
// Not currently supported // Not currently supported
break; break;
@ -2181,24 +2025,6 @@ public class Controller extends Thread {
return now - offset; return now - offset;
} }
private static void cleanChunkUploadTempDir() {
Path uploadsTemp = Paths.get("uploads-temp");
if (!Files.exists(uploadsTemp)) {
return;
}
try (Stream<Path> paths = Files.walk(uploadsTemp)) {
paths.sorted(Comparator.reverseOrder())
.map(Path::toFile)
.forEach(File::delete);
LOGGER.info("Cleaned up all temporary uploads in {}", uploadsTemp);
} catch (IOException e) {
LOGGER.warn("Failed to clean up uploads-temp directory", e);
}
}
public StatsSnapshot getStatsSnapshot() { public StatsSnapshot getStatsSnapshot() {
return this.stats; return this.stats;
} }

File diff suppressed because it is too large Load Diff

View File

@ -13,7 +13,6 @@ import org.qortal.crypto.MemoryPoW;
import org.qortal.crypto.Qortal25519Extras; import org.qortal.crypto.Qortal25519Extras;
import org.qortal.data.account.MintingAccountData; import org.qortal.data.account.MintingAccountData;
import org.qortal.data.account.RewardShareData; import org.qortal.data.account.RewardShareData;
import org.qortal.data.group.GroupMemberData;
import org.qortal.data.network.OnlineAccountData; import org.qortal.data.network.OnlineAccountData;
import org.qortal.network.Network; import org.qortal.network.Network;
import org.qortal.network.Peer; import org.qortal.network.Peer;
@ -25,7 +24,6 @@ import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager; import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import org.qortal.utils.Groups;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import org.qortal.utils.NamedThreadFactory; import org.qortal.utils.NamedThreadFactory;
@ -46,7 +44,6 @@ public class OnlineAccountsManager {
*/ */
private static final long ONLINE_TIMESTAMP_MODULUS_V1 = 5 * 60 * 1000L; private static final long ONLINE_TIMESTAMP_MODULUS_V1 = 5 * 60 * 1000L;
private static final long ONLINE_TIMESTAMP_MODULUS_V2 = 30 * 60 * 1000L; private static final long ONLINE_TIMESTAMP_MODULUS_V2 = 30 * 60 * 1000L;
private static final long ONLINE_TIMESTAMP_MODULUS_V3 = 10 * 60 * 1000L;
/** /**
* How many 'current' timestamp-sets of online accounts we cache. * How many 'current' timestamp-sets of online accounts we cache.
@ -70,13 +67,12 @@ public class OnlineAccountsManager {
private static final long ONLINE_ACCOUNTS_COMPUTE_INITIAL_SLEEP_INTERVAL = 30 * 1000L; // ms private static final long ONLINE_ACCOUNTS_COMPUTE_INITIAL_SLEEP_INTERVAL = 30 * 1000L; // ms
// MemoryPoW - mainnet // MemoryPoW - mainnet
public static final int POW_BUFFER_SIZE = 1024 * 1024; // bytes public static final int POW_BUFFER_SIZE = 1 * 1024 * 1024; // bytes
public static final int POW_DIFFICULTY_V1 = 18; // leading zero bits public static final int POW_DIFFICULTY_V1 = 18; // leading zero bits
public static final int POW_DIFFICULTY_V2 = 19; // leading zero bits public static final int POW_DIFFICULTY_V2 = 19; // leading zero bits
public static final int POW_DIFFICULTY_V3 = 6; // leading zero bits
// MemoryPoW - testnet // MemoryPoW - testnet
public static final int POW_BUFFER_SIZE_TESTNET = 1024 * 1024; // bytes public static final int POW_BUFFER_SIZE_TESTNET = 1 * 1024 * 1024; // bytes
public static final int POW_DIFFICULTY_TESTNET = 5; // leading zero bits public static final int POW_DIFFICULTY_TESTNET = 5; // leading zero bits
// IMPORTANT: if we ever need to dynamically modify the buffer size using a feature trigger, the // IMPORTANT: if we ever need to dynamically modify the buffer size using a feature trigger, the
@ -84,7 +80,7 @@ public class OnlineAccountsManager {
// one for the transition period. // one for the transition period.
private static long[] POW_VERIFY_WORK_BUFFER = new long[getPoWBufferSize() / 8]; private static long[] POW_VERIFY_WORK_BUFFER = new long[getPoWBufferSize() / 8];
private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("OnlineAccounts", Thread.NORM_PRIORITY)); private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("OnlineAccounts"));
private volatile boolean isStopping = false; private volatile boolean isStopping = false;
private final Set<OnlineAccountData> onlineAccountsImportQueue = ConcurrentHashMap.newKeySet(); private final Set<OnlineAccountData> onlineAccountsImportQueue = ConcurrentHashMap.newKeySet();
@ -110,15 +106,11 @@ public class OnlineAccountsManager {
public static long getOnlineTimestampModulus() { public static long getOnlineTimestampModulus() {
Long now = NTP.getTime(); Long now = NTP.getTime();
if (now != null && now >= BlockChain.getInstance().getOnlineAccountsModulusV2Timestamp() && now < BlockChain.getInstance().getOnlineAccountsModulusV3Timestamp()) { if (now != null && now >= BlockChain.getInstance().getOnlineAccountsModulusV2Timestamp()) {
return ONLINE_TIMESTAMP_MODULUS_V2; return ONLINE_TIMESTAMP_MODULUS_V2;
} }
if (now != null && now >= BlockChain.getInstance().getOnlineAccountsModulusV3Timestamp()) {
return ONLINE_TIMESTAMP_MODULUS_V3;
}
return ONLINE_TIMESTAMP_MODULUS_V1; return ONLINE_TIMESTAMP_MODULUS_V1;
} }
public static Long getCurrentOnlineAccountTimestamp() { public static Long getCurrentOnlineAccountTimestamp() {
Long now = NTP.getTime(); Long now = NTP.getTime();
if (now == null) if (now == null)
@ -143,12 +135,9 @@ public class OnlineAccountsManager {
if (Settings.getInstance().isTestNet()) if (Settings.getInstance().isTestNet())
return POW_DIFFICULTY_TESTNET; return POW_DIFFICULTY_TESTNET;
if (timestamp >= BlockChain.getInstance().getIncreaseOnlineAccountsDifficultyTimestamp() && timestamp < BlockChain.getInstance().getDecreaseOnlineAccountsDifficultyTimestamp()) if (timestamp >= BlockChain.getInstance().getIncreaseOnlineAccountsDifficultyTimestamp())
return POW_DIFFICULTY_V2; return POW_DIFFICULTY_V2;
if (timestamp >= BlockChain.getInstance().getDecreaseOnlineAccountsDifficultyTimestamp())
return POW_DIFFICULTY_V3;
return POW_DIFFICULTY_V1; return POW_DIFFICULTY_V1;
} }
@ -226,15 +215,6 @@ public class OnlineAccountsManager {
Set<OnlineAccountData> onlineAccountsToAdd = new HashSet<>(); Set<OnlineAccountData> onlineAccountsToAdd = new HashSet<>();
Set<OnlineAccountData> onlineAccountsToRemove = new HashSet<>(); Set<OnlineAccountData> onlineAccountsToRemove = new HashSet<>();
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
int blockHeight = repository.getBlockRepository().getBlockchainHeight();
List<String> mintingGroupMemberAddresses
= Groups.getAllMembers(
repository.getGroupRepository(),
Groups.getGroupIdsToMint(BlockChain.getInstance(), blockHeight)
);
for (OnlineAccountData onlineAccountData : this.onlineAccountsImportQueue) { for (OnlineAccountData onlineAccountData : this.onlineAccountsImportQueue) {
if (isStopping) if (isStopping)
return; return;
@ -247,7 +227,7 @@ public class OnlineAccountsManager {
continue; continue;
} }
boolean isValid = this.isValidCurrentAccount(repository, mintingGroupMemberAddresses, onlineAccountData); boolean isValid = this.isValidCurrentAccount(repository, onlineAccountData);
if (isValid) if (isValid)
onlineAccountsToAdd.add(onlineAccountData); onlineAccountsToAdd.add(onlineAccountData);
@ -326,7 +306,7 @@ public class OnlineAccountsManager {
return inplaceArray; return inplaceArray;
} }
private static boolean isValidCurrentAccount(Repository repository, List<String> mintingGroupMemberAddresses, OnlineAccountData onlineAccountData) throws DataException { private static boolean isValidCurrentAccount(Repository repository, OnlineAccountData onlineAccountData) throws DataException {
final Long now = NTP.getTime(); final Long now = NTP.getTime();
if (now == null) if (now == null)
return false; return false;
@ -361,14 +341,9 @@ public class OnlineAccountsManager {
LOGGER.trace(() -> String.format("Rejecting unknown online reward-share public key %s", Base58.encode(rewardSharePublicKey))); LOGGER.trace(() -> String.format("Rejecting unknown online reward-share public key %s", Base58.encode(rewardSharePublicKey)));
return false; return false;
} }
// reject account address that are not in the MINTER Group
else if( !mintingGroupMemberAddresses.contains(rewardShareData.getMinter())) {
LOGGER.trace(() -> String.format("Rejecting online reward-share that is not in MINTER Group, account %s", rewardShareData.getMinter()));
return false;
}
Account mintingAccount = new Account(repository, rewardShareData.getMinter()); Account mintingAccount = new Account(repository, rewardShareData.getMinter());
if (!mintingAccount.canMint(true)) { // group validation is a few lines above if (!mintingAccount.canMint()) {
// Minting-account component of reward-share can no longer mint - disregard // Minting-account component of reward-share can no longer mint - disregard
LOGGER.trace(() -> String.format("Rejecting online reward-share with non-minting account %s", mintingAccount.getAddress())); LOGGER.trace(() -> String.format("Rejecting online reward-share with non-minting account %s", mintingAccount.getAddress()));
return false; return false;
@ -555,7 +530,7 @@ public class OnlineAccountsManager {
} }
Account mintingAccount = new Account(repository, rewardShareData.getMinter()); Account mintingAccount = new Account(repository, rewardShareData.getMinter());
if (!mintingAccount.canMint(true)) { if (!mintingAccount.canMint()) {
// Minting-account component of reward-share can no longer mint - disregard // Minting-account component of reward-share can no longer mint - disregard
iterator.remove(); iterator.remove();
continue; continue;

View File

@ -65,7 +65,6 @@ public class PirateChainWalletController extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Pirate Chain Wallet Controller"); Thread.currentThread().setName("Pirate Chain Wallet Controller");
Thread.currentThread().setPriority(MIN_PRIORITY);
try { try {
while (running && !Controller.isStopping()) { while (running && !Controller.isStopping()) {

View File

@ -118,12 +118,8 @@ public class Synchronizer extends Thread {
} }
public static Synchronizer getInstance() { public static Synchronizer getInstance() {
if (instance == null) { if (instance == null)
instance = new Synchronizer(); instance = new Synchronizer();
instance.setPriority(Settings.getInstance().getSynchronizerThreadPriority());
LOGGER.info("thread priority = " + instance.getPriority());
}
return instance; return instance;
} }

View File

@ -2,7 +2,6 @@ package org.qortal.controller;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.qortal.controller.arbitrary.PeerMessage;
import org.qortal.data.block.BlockData; import org.qortal.data.block.BlockData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
import org.qortal.network.Network; import org.qortal.network.Network;
@ -21,11 +20,7 @@ import org.qortal.utils.Base58;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import java.util.*; import java.util.*;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock; import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Function;
import java.util.stream.Collectors; import java.util.stream.Collectors;
public class TransactionImporter extends Thread { public class TransactionImporter extends Thread {
@ -55,10 +50,6 @@ public class TransactionImporter extends Thread {
/** Cached list of unconfirmed transactions, used when counting per creator. This is replaced regularly */ /** Cached list of unconfirmed transactions, used when counting per creator. This is replaced regularly */
public static List<TransactionData> unconfirmedTransactionsCache = null; public static List<TransactionData> unconfirmedTransactionsCache = null;
public TransactionImporter() {
signatureMessageScheduler.scheduleAtFixedRate(this::processNetworkTransactionSignaturesMessage, 60, 1, TimeUnit.SECONDS);
getTransactionMessageScheduler.scheduleAtFixedRate(this::processNetworkGetTransactionMessages, 60, 1, TimeUnit.SECONDS);
}
public static synchronized TransactionImporter getInstance() { public static synchronized TransactionImporter getInstance() {
if (instance == null) { if (instance == null) {
@ -380,104 +371,36 @@ public class TransactionImporter extends Thread {
} }
} }
// List to collect messages
private final List<PeerMessage> getTransactionMessageList = new ArrayList<>();
// Lock to synchronize access to the list
private final Object getTransactionMessageLock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService getTransactionMessageScheduler = Executors.newScheduledThreadPool(1);
public void onNetworkGetTransactionMessage(Peer peer, Message message) { public void onNetworkGetTransactionMessage(Peer peer, Message message) {
GetTransactionMessage getTransactionMessage = (GetTransactionMessage) message;
byte[] signature = getTransactionMessage.getSignature();
synchronized (getTransactionMessageLock) { try (final Repository repository = RepositoryManager.getRepository()) {
getTransactionMessageList.add(new PeerMessage(peer, message));
}
}
private void processNetworkGetTransactionMessages() {
try {
List<PeerMessage> messagesToProcess;
synchronized (getTransactionMessageLock) {
messagesToProcess = new ArrayList<>(getTransactionMessageList);
getTransactionMessageList.clear();
}
if( messagesToProcess.isEmpty() ) return;
Map<String, PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
for( PeerMessage peerMessage : messagesToProcess ) {
GetTransactionMessage getTransactionMessage = (GetTransactionMessage) peerMessage.getMessage();
byte[] signature = getTransactionMessage.getSignature();
peerMessageBySignature58.put(Base58.encode(signature), peerMessage);
}
// Firstly check the sig-valid transactions that are currently queued for import // Firstly check the sig-valid transactions that are currently queued for import
Map<String, TransactionData> transactionsCachedBySignature58 TransactionData transactionData = this.getCachedSigValidTransactions().stream()
= this.getCachedSigValidTransactions().stream() .filter(t -> Arrays.equals(signature, t.getSignature()))
.collect(Collectors.toMap(t -> Base58.encode(t.getSignature()), Function.identity())); .findFirst().orElse(null);
Map<Boolean, List<Map.Entry<String, PeerMessage>>> transactionsCachedBySignature58Partition if (transactionData == null) {
= peerMessageBySignature58.entrySet().stream()
.collect(Collectors.partitioningBy(entry -> transactionsCachedBySignature58.containsKey(entry.getKey())));
List<byte[]> signaturesNeeded
= transactionsCachedBySignature58Partition.get(false).stream()
.map(Map.Entry::getValue)
.map(PeerMessage::getMessage)
.map(message -> (GetTransactionMessage) message)
.map(GetTransactionMessage::getSignature)
.collect(Collectors.toList());
// transaction found in the import queue
Map<String, TransactionData> transactionsToSendBySignature58 = new HashMap<>(messagesToProcess.size());
for( Map.Entry<String, PeerMessage> entry : transactionsCachedBySignature58Partition.get(true)) {
transactionsToSendBySignature58.put(entry.getKey(), transactionsCachedBySignature58.get(entry.getKey()));
}
if( !signaturesNeeded.isEmpty() ) {
// Not found in import queue, so try the database // Not found in import queue, so try the database
try (final Repository repository = RepositoryManager.getRepository()) { transactionData = repository.getTransactionRepository().fromSignature(signature);
transactionsToSendBySignature58.putAll(
repository.getTransactionRepository().fromSignatures(signaturesNeeded).stream()
.collect(Collectors.toMap(transactionData -> Base58.encode(transactionData.getSignature()), Function.identity()))
);
} catch (DataException e) {
LOGGER.error(e.getMessage(), e);
}
} }
for( final Map.Entry<String, TransactionData> entry : transactionsToSendBySignature58.entrySet() ) { if (transactionData == null) {
// Still not found - so we don't have this transaction
PeerMessage peerMessage = peerMessageBySignature58.get(entry.getKey()); LOGGER.debug(() -> String.format("Ignoring GET_TRANSACTION request from peer %s for unknown transaction %s", peer, Base58.encode(signature)));
final Message message = peerMessage.getMessage(); // Send no response at all???
final Peer peer = peerMessage.getPeer(); return;
Runnable sendTransactionMessageRunner = () -> sendTransactionMessage(entry.getKey(), entry.getValue(), message, peer);
Thread sendTransactionMessageThread = new Thread(sendTransactionMessageRunner);
sendTransactionMessageThread.start();
} }
} catch (Exception e) {
LOGGER.error(e.getMessage(),e);
}
}
private static void sendTransactionMessage(String signature58, TransactionData data, Message message, Peer peer) { Message transactionMessage = new TransactionMessage(transactionData);
try {
Message transactionMessage = new TransactionMessage(data);
transactionMessage.setId(message.getId()); transactionMessage.setId(message.getId());
if (!peer.sendMessage(transactionMessage)) if (!peer.sendMessage(transactionMessage))
peer.disconnect("failed to send transaction"); peer.disconnect("failed to send transaction");
} } catch (DataException e) {
catch (TransformationException e) { LOGGER.error(String.format("Repository issue while sending transaction %s to peer %s", Base58.encode(signature), peer), e);
LOGGER.error(String.format("Serialization issue while sending transaction %s to peer %s", signature58, peer), e); } catch (TransformationException e) {
} LOGGER.error(String.format("Serialization issue while sending transaction %s to peer %s", Base58.encode(signature), peer), e);
catch (Exception e) {
LOGGER.error(e.getMessage(), e);
} }
} }
@ -498,86 +421,44 @@ public class TransactionImporter extends Thread {
} }
} }
// List to collect messages
private final List<PeerMessage> signatureMessageList = new ArrayList<>();
// Lock to synchronize access to the list
private final Object signatureMessageLock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService signatureMessageScheduler = Executors.newScheduledThreadPool(1);
public void onNetworkTransactionSignaturesMessage(Peer peer, Message message) { public void onNetworkTransactionSignaturesMessage(Peer peer, Message message) {
synchronized (signatureMessageLock) { TransactionSignaturesMessage transactionSignaturesMessage = (TransactionSignaturesMessage) message;
signatureMessageList.add(new PeerMessage(peer, message)); List<byte[]> signatures = transactionSignaturesMessage.getSignatures();
}
}
public void processNetworkTransactionSignaturesMessage() { try (final Repository repository = RepositoryManager.getRepository()) {
for (byte[] signature : signatures) {
try { String signature58 = Base58.encode(signature);
List<PeerMessage> messagesToProcess; if (invalidUnconfirmedTransactions.containsKey(signature58)) {
synchronized (signatureMessageLock) { // Previously invalid transaction - don't keep requesting it
messagesToProcess = new ArrayList<>(signatureMessageList); // It will be periodically removed from invalidUnconfirmedTransactions to allow for rechecks
signatureMessageList.clear(); continue;
}
Map<String, byte[]> signatureBySignature58 = new HashMap<>(messagesToProcess.size() * 10);
Map<String, Peer> peerBySignature58 = new HashMap<>( messagesToProcess.size() * 10 );
for( PeerMessage peerMessage : messagesToProcess ) {
TransactionSignaturesMessage transactionSignaturesMessage = (TransactionSignaturesMessage) peerMessage.getMessage();
List<byte[]> signatures = transactionSignaturesMessage.getSignatures();
for (byte[] signature : signatures) {
String signature58 = Base58.encode(signature);
if (invalidUnconfirmedTransactions.containsKey(signature58)) {
// Previously invalid transaction - don't keep requesting it
// It will be periodically removed from invalidUnconfirmedTransactions to allow for rechecks
continue;
}
// Ignore if this transaction is in the queue
if (incomingTransactionQueueContains(signature)) {
LOGGER.trace(() -> String.format("Ignoring existing queued transaction %s from peer %s", Base58.encode(signature), peerMessage.getPeer()));
continue;
}
signatureBySignature58.put(signature58, signature);
peerBySignature58.put(signature58, peerMessage.getPeer());
} }
}
if( !signatureBySignature58.isEmpty() ) { // Ignore if this transaction is in the queue
try (final Repository repository = RepositoryManager.getRepository()) { if (incomingTransactionQueueContains(signature)) {
LOGGER.trace(() -> String.format("Ignoring existing queued transaction %s from peer %s", Base58.encode(signature), peer));
// remove signatures in db already continue;
repository.getTransactionRepository()
.fromSignatures(new ArrayList<>(signatureBySignature58.values())).stream()
.map(TransactionData::getSignature)
.map(signature -> Base58.encode(signature))
.forEach(signature58 -> signatureBySignature58.remove(signature58));
} catch (DataException e) {
LOGGER.error(String.format("Repository issue while processing unconfirmed transactions from peer"), e);
} }
}
// Check isInterrupted() here and exit fast // Do we have it already? (Before requesting transaction data itself)
if (Thread.currentThread().isInterrupted()) if (repository.getTransactionRepository().exists(signature)) {
return; LOGGER.trace(() -> String.format("Ignoring existing transaction %s from peer %s", Base58.encode(signature), peer));
continue;
}
for (Map.Entry<String, byte[]> entry : signatureBySignature58.entrySet()) { // Check isInterrupted() here and exit fast
if (Thread.currentThread().isInterrupted())
Peer peer = peerBySignature58.get(entry.getKey()); return;
// Fetch actual transaction data from peer // Fetch actual transaction data from peer
Message getTransactionMessage = new GetTransactionMessage(entry.getValue()); Message getTransactionMessage = new GetTransactionMessage(signature);
if (peer != null && !peer.sendMessage(getTransactionMessage)) { if (!peer.sendMessage(getTransactionMessage)) {
peer.disconnect("failed to request transaction"); peer.disconnect("failed to request transaction");
return;
} }
} }
} catch (Exception e) { } catch (DataException e) {
LOGGER.error(e.getMessage(), e); LOGGER.error(String.format("Repository issue while processing unconfirmed transactions from peer %s", peer), e);
} }
} }

View File

@ -14,7 +14,6 @@ import java.io.IOException;
import java.util.Comparator; import java.util.Comparator;
import java.util.Map; import java.util.Map;
import static java.lang.Thread.NORM_PRIORITY;
import static org.qortal.data.arbitrary.ArbitraryResourceStatus.Status.NOT_PUBLISHED; import static org.qortal.data.arbitrary.ArbitraryResourceStatus.Status.NOT_PUBLISHED;
@ -29,7 +28,6 @@ public class ArbitraryDataBuilderThread implements Runnable {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data Builder Thread"); Thread.currentThread().setName("Arbitrary Data Builder Thread");
Thread.currentThread().setPriority(NORM_PRIORITY);
ArbitraryDataBuildManager buildManager = ArbitraryDataBuildManager.getInstance(); ArbitraryDataBuildManager buildManager = ArbitraryDataBuildManager.getInstance();
while (!Controller.isStopping()) { while (!Controller.isStopping()) {

View File

@ -2,30 +2,22 @@ package org.qortal.controller.arbitrary;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.qortal.api.resource.TransactionsResource;
import org.qortal.controller.Controller; import org.qortal.controller.Controller;
import org.qortal.data.arbitrary.ArbitraryResourceData; import org.qortal.data.arbitrary.ArbitraryResourceData;
import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.event.DataMonitorEvent;
import org.qortal.event.EventBus;
import org.qortal.gui.SplashFrame; import org.qortal.gui.SplashFrame;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
import org.qortal.repository.Repository; import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager; import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.transaction.ArbitraryTransaction; import org.qortal.transaction.ArbitraryTransaction;
import org.qortal.transaction.Transaction;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import java.text.NumberFormat;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Collections; import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.function.Function;
import java.util.stream.Collectors;
public class ArbitraryDataCacheManager extends Thread { public class ArbitraryDataCacheManager extends Thread {
@ -37,11 +29,6 @@ public class ArbitraryDataCacheManager extends Thread {
/** Queue of arbitrary transactions that require cache updates */ /** Queue of arbitrary transactions that require cache updates */
private final List<ArbitraryTransactionData> updateQueue = Collections.synchronizedList(new ArrayList<>()); private final List<ArbitraryTransactionData> updateQueue = Collections.synchronizedList(new ArrayList<>());
private static final NumberFormat FORMATTER = NumberFormat.getNumberInstance();
static {
FORMATTER.setGroupingUsed(true);
}
public static synchronized ArbitraryDataCacheManager getInstance() { public static synchronized ArbitraryDataCacheManager getInstance() {
if (instance == null) { if (instance == null) {
@ -54,26 +41,20 @@ public class ArbitraryDataCacheManager extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data Cache Manager"); Thread.currentThread().setName("Arbitrary Data Cache Manager");
Thread.currentThread().setPriority(NORM_PRIORITY);
try { try {
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
try { Thread.sleep(500L);
Thread.sleep(500L);
// Process queue // Process queue
processResourceQueue(); processResourceQueue();
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
Thread.sleep(600_000L); // wait 10 minutes to continue
}
} }
} catch (InterruptedException e) {
// Clear queue before terminating thread // Fall through to exit thread
processResourceQueue();
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
} }
// Clear queue before terminating thread
processResourceQueue();
} }
public void shutdown() { public void shutdown() {
@ -103,25 +84,14 @@ public class ArbitraryDataCacheManager extends Thread {
// Update arbitrary resource caches // Update arbitrary resource caches
try { try {
ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData); ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
arbitraryTransaction.updateArbitraryResourceCacheIncludingMetadata(repository, new HashSet<>(0), new HashMap<>(0)); arbitraryTransaction.updateArbitraryResourceCache(repository);
arbitraryTransaction.updateArbitraryMetadataCache(repository);
repository.saveChanges(); repository.saveChanges();
// Update status as separate commit, as this is more prone to failure // Update status as separate commit, as this is more prone to failure
arbitraryTransaction.updateArbitraryResourceStatus(repository); arbitraryTransaction.updateArbitraryResourceStatus(repository);
repository.saveChanges(); repository.saveChanges();
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
transactionData.getIdentifier(),
transactionData.getName(),
transactionData.getService().name(),
"updated resource cache and status, queue",
transactionData.getTimestamp(),
transactionData.getTimestamp()
)
);
LOGGER.debug(() -> String.format("Finished processing transaction %.8s in arbitrary resource queue...", Base58.encode(transactionData.getSignature()))); LOGGER.debug(() -> String.format("Finished processing transaction %.8s in arbitrary resource queue...", Base58.encode(transactionData.getSignature())));
} catch (DataException e) { } catch (DataException e) {
@ -132,9 +102,6 @@ public class ArbitraryDataCacheManager extends Thread {
} catch (DataException e) { } catch (DataException e) {
LOGGER.error("Repository issue while processing arbitrary resource cache updates", e); LOGGER.error("Repository issue while processing arbitrary resource cache updates", e);
} }
catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
} }
public void addToUpdateQueue(ArbitraryTransactionData transactionData) { public void addToUpdateQueue(ArbitraryTransactionData transactionData) {
@ -180,66 +147,34 @@ public class ArbitraryDataCacheManager extends Thread {
LOGGER.info("Building arbitrary resources cache..."); LOGGER.info("Building arbitrary resources cache...");
SplashFrame.getInstance().updateStatus("Building QDN cache - please wait..."); SplashFrame.getInstance().updateStatus("Building QDN cache - please wait...");
final int batchSize = Settings.getInstance().getBuildArbitraryResourcesBatchSize(); final int batchSize = 100;
int offset = 0; int offset = 0;
List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder
= repository.getArbitraryRepository().getLatestArbitraryTransactions();
LOGGER.info("arbitrary transactions: count = " + allArbitraryTransactionsInDescendingOrder.size());
List<ArbitraryResourceData> resources = repository.getArbitraryRepository().getArbitraryResources(null, null, true);
Map<ArbitraryTransactionDataHashWrapper, ArbitraryResourceData> resourceByWrapper = new HashMap<>(resources.size());
for( ArbitraryResourceData resource : resources ) {
resourceByWrapper.put(
new ArbitraryTransactionDataHashWrapper(resource.service.value, resource.name, resource.identifier),
resource
);
}
LOGGER.info("arbitrary resources: count = " + resourceByWrapper.size());
Set<ArbitraryTransactionDataHashWrapper> latestTransactionsWrapped = new HashSet<>(allArbitraryTransactionsInDescendingOrder.size());
// Loop through all ARBITRARY transactions, and determine latest state // Loop through all ARBITRARY transactions, and determine latest state
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
LOGGER.info( LOGGER.info("Fetching arbitrary transactions {} - {}", offset, offset+batchSize-1);
"Fetching arbitrary transactions {} - {} / {} Total",
FORMATTER.format(offset),
FORMATTER.format(offset+batchSize-1),
FORMATTER.format(allArbitraryTransactionsInDescendingOrder.size())
);
List<ArbitraryTransactionData> transactionsToProcess List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, List.of(Transaction.TransactionType.ARBITRARY), null, null, null, TransactionsResource.ConfirmationStatus.BOTH, batchSize, offset, false);
= allArbitraryTransactionsInDescendingOrder.stream() if (signatures.isEmpty()) {
.skip(offset)
.limit(batchSize)
.collect(Collectors.toList());
if (transactionsToProcess.isEmpty()) {
// Complete // Complete
break; break;
} }
try { // Expand signatures to transactions
for( ArbitraryTransactionData transactionData : transactionsToProcess) { for (byte[] signature : signatures) {
if (transactionData.getService() == null) { ArbitraryTransactionData transactionData = (ArbitraryTransactionData) repository
// Unsupported service - ignore this resource .getTransactionRepository().fromSignature(signature);
continue;
}
latestTransactionsWrapped.add(new ArbitraryTransactionDataHashWrapper(transactionData)); if (transactionData.getService() == null) {
// Unsupported service - ignore this resource
// Update arbitrary resource caches continue;
ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
arbitraryTransaction.updateArbitraryResourceCacheIncludingMetadata(repository, latestTransactionsWrapped, resourceByWrapper);
} }
repository.saveChanges();
} catch (DataException e) {
repository.discardChanges();
LOGGER.error(e.getMessage(), e); // Update arbitrary resource caches
ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
arbitraryTransaction.updateArbitraryResourceCache(repository);
arbitraryTransaction.updateArbitraryMetadataCache(repository);
repository.saveChanges();
} }
offset += batchSize; offset += batchSize;
} }
@ -257,11 +192,6 @@ public class ArbitraryDataCacheManager extends Thread {
repository.discardChanges(); repository.discardChanges();
throw new DataException("Build of arbitrary resources cache failed."); throw new DataException("Build of arbitrary resources cache failed.");
} }
catch (Exception e) {
LOGGER.error(e.getMessage(), e);
return false;
}
} }
private boolean refreshArbitraryStatuses(Repository repository) throws DataException { private boolean refreshArbitraryStatuses(Repository repository) throws DataException {
@ -269,48 +199,27 @@ public class ArbitraryDataCacheManager extends Thread {
LOGGER.info("Refreshing arbitrary resource statuses for locally hosted transactions..."); LOGGER.info("Refreshing arbitrary resource statuses for locally hosted transactions...");
SplashFrame.getInstance().updateStatus("Refreshing statuses - please wait..."); SplashFrame.getInstance().updateStatus("Refreshing statuses - please wait...");
final int batchSize = Settings.getInstance().getBuildArbitraryResourcesBatchSize(); final int batchSize = 100;
int offset = 0; int offset = 0;
List<ArbitraryTransactionData> allHostedTransactions
= ArbitraryDataStorageManager.getInstance()
.listAllHostedTransactions(repository, null, null);
// Loop through all ARBITRARY transactions, and determine latest state // Loop through all ARBITRARY transactions, and determine latest state
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
LOGGER.info( LOGGER.info("Fetching hosted transactions {} - {}", offset, offset+batchSize-1);
"Fetching hosted transactions {} - {} / {} Total",
FORMATTER.format(offset),
FORMATTER.format(offset+batchSize-1),
FORMATTER.format(allHostedTransactions.size())
);
List<ArbitraryTransactionData> hostedTransactions
= allHostedTransactions.stream()
.skip(offset)
.limit(batchSize)
.collect(Collectors.toList());
List<ArbitraryTransactionData> hostedTransactions = ArbitraryDataStorageManager.getInstance().listAllHostedTransactions(repository, batchSize, offset);
if (hostedTransactions.isEmpty()) { if (hostedTransactions.isEmpty()) {
// Complete // Complete
break; break;
} }
try { // Loop through hosted transactions
// Loop through hosted transactions for (ArbitraryTransactionData transactionData : hostedTransactions) {
for (ArbitraryTransactionData transactionData : hostedTransactions) {
// Determine status and update cache // Determine status and update cache
ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData); ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
arbitraryTransaction.updateArbitraryResourceStatus(repository); arbitraryTransaction.updateArbitraryResourceStatus(repository);
}
repository.saveChanges(); repository.saveChanges();
} catch (DataException e) {
repository.discardChanges();
LOGGER.error(e.getMessage(), e);
} }
offset += batchSize; offset += batchSize;
} }
@ -324,11 +233,6 @@ public class ArbitraryDataCacheManager extends Thread {
repository.discardChanges(); repository.discardChanges();
throw new DataException("Refresh of arbitrary resource statuses failed."); throw new DataException("Refresh of arbitrary resource statuses failed.");
} }
catch (Exception e) {
LOGGER.error(e.getMessage(), e);
return false;
}
} }
} }

View File

@ -2,10 +2,9 @@ package org.qortal.controller.arbitrary;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.qortal.api.resource.TransactionsResource.ConfirmationStatus;
import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
import org.qortal.event.DataMonitorEvent;
import org.qortal.event.EventBus;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
import org.qortal.repository.Repository; import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager; import org.qortal.repository.RepositoryManager;
@ -22,12 +21,8 @@ import java.nio.file.Paths;
import java.security.SecureRandom; import java.security.SecureRandom;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Arrays; import java.util.Arrays;
import java.util.HashSet;
import java.util.List; import java.util.List;
import java.util.Objects; import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
import static org.qortal.controller.arbitrary.ArbitraryDataStorageManager.DELETION_THRESHOLD; import static org.qortal.controller.arbitrary.ArbitraryDataStorageManager.DELETION_THRESHOLD;
@ -76,25 +71,11 @@ public class ArbitraryDataCleanupManager extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data Cleanup Manager"); Thread.currentThread().setName("Arbitrary Data Cleanup Manager");
Thread.currentThread().setPriority(NORM_PRIORITY);
// Paginate queries when fetching arbitrary transactions // Paginate queries when fetching arbitrary transactions
final int limit = 100; final int limit = 100;
int offset = 0; int offset = 0;
List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder;
try (final Repository repository = RepositoryManager.getRepository()) {
allArbitraryTransactionsInDescendingOrder
= repository.getArbitraryRepository()
.getLatestArbitraryTransactions();
} catch( Exception e) {
LOGGER.error(e.getMessage(), e);
allArbitraryTransactionsInDescendingOrder = new ArrayList<>(0);
}
Set<ArbitraryTransactionData> processedTransactions = new HashSet<>();
try { try {
while (!isStopping) { while (!isStopping) {
Thread.sleep(30000); Thread.sleep(30000);
@ -125,31 +106,27 @@ public class ArbitraryDataCleanupManager extends Thread {
// Any arbitrary transactions we want to fetch data for? // Any arbitrary transactions we want to fetch data for?
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
List<ArbitraryTransactionData> transactions = allArbitraryTransactionsInDescendingOrder.stream().skip(offset).limit(limit).collect(Collectors.toList()); List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, ARBITRARY_TX_TYPE, null, null, null, ConfirmationStatus.BOTH, limit, offset, true);
// LOGGER.info("Found {} arbitrary transactions at offset: {}, limit: {}", signatures.size(), offset, limit);
if (isStopping) { if (isStopping) {
return; return;
} }
if (transactions == null || transactions.isEmpty()) { if (signatures == null || signatures.isEmpty()) {
offset = 0; offset = 0;
allArbitraryTransactionsInDescendingOrder continue;
= repository.getArbitraryRepository()
.getLatestArbitraryTransactions();
transactions = allArbitraryTransactionsInDescendingOrder.stream().limit(limit).collect(Collectors.toList());
processedTransactions.clear();
} }
offset += limit; offset += limit;
now = NTP.getTime(); now = NTP.getTime();
// Loop through the signatures in this batch // Loop through the signatures in this batch
for (int i=0; i<transactions.size(); i++) { for (int i=0; i<signatures.size(); i++) {
if (isStopping) { if (isStopping) {
return; return;
} }
ArbitraryTransactionData arbitraryTransactionData = transactions.get(i); byte[] signature = signatures.get(i);
if (arbitraryTransactionData == null) { if (signature == null) {
continue; continue;
} }
@ -158,7 +135,9 @@ public class ArbitraryDataCleanupManager extends Thread {
Thread.sleep(5000); Thread.sleep(5000);
} }
if (arbitraryTransactionData.getService() == null) { // Fetch the transaction data
ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
if (arbitraryTransactionData == null || arbitraryTransactionData.getService() == null) {
continue; continue;
} }
@ -167,8 +146,6 @@ public class ArbitraryDataCleanupManager extends Thread {
continue; continue;
} }
boolean mostRecentTransaction = processedTransactions.add(arbitraryTransactionData);
// Check if we have the complete file // Check if we have the complete file
boolean completeFileExists = ArbitraryTransactionUtils.completeFileExists(arbitraryTransactionData); boolean completeFileExists = ArbitraryTransactionUtils.completeFileExists(arbitraryTransactionData);
@ -189,54 +166,20 @@ public class ArbitraryDataCleanupManager extends Thread {
LOGGER.info("Deleting transaction {} because we can't host its data", LOGGER.info("Deleting transaction {} because we can't host its data",
Base58.encode(arbitraryTransactionData.getSignature())); Base58.encode(arbitraryTransactionData.getSignature()));
ArbitraryTransactionUtils.deleteCompleteFileAndChunks(arbitraryTransactionData); ArbitraryTransactionUtils.deleteCompleteFileAndChunks(arbitraryTransactionData);
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"can't store data, deleting",
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
continue; continue;
} }
// Check to see if we have had a more recent PUT // Check to see if we have had a more recent PUT
if (!mostRecentTransaction) { boolean hasMoreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
if (hasMoreRecentPutTransaction) {
// There is a more recent PUT transaction than the one we are currently processing. // There is a more recent PUT transaction than the one we are currently processing.
// When a PUT is issued, it replaces any layers that would have been there before. // When a PUT is issued, it replaces any layers that would have been there before.
// Therefore any data relating to this older transaction is no longer needed. // Therefore any data relating to this older transaction is no longer needed.
LOGGER.info(String.format("Newer PUT found for %s %s since transaction %s. " + LOGGER.info(String.format("Newer PUT found for %s %s since transaction %s. " +
"Deleting all files associated with the earlier transaction.", arbitraryTransactionData.getService(), "Deleting all files associated with the earlier transaction.", arbitraryTransactionData.getService(),
arbitraryTransactionData.getName(), Base58.encode(arbitraryTransactionData.getSignature()))); arbitraryTransactionData.getName(), Base58.encode(signature)));
ArbitraryTransactionUtils.deleteCompleteFileAndChunks(arbitraryTransactionData); ArbitraryTransactionUtils.deleteCompleteFileAndChunks(arbitraryTransactionData);
Optional<ArbitraryTransactionData> moreRecentPutTransaction
= processedTransactions.stream()
.filter(data -> data.equals(arbitraryTransactionData))
.findAny();
if( moreRecentPutTransaction.isPresent() ) {
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"deleting data due to replacement",
arbitraryTransactionData.getTimestamp(),
moreRecentPutTransaction.get().getTimestamp()
)
);
}
else {
LOGGER.warn("Something went wrong with the most recent put transaction determination!");
}
continue; continue;
} }
@ -255,21 +198,7 @@ public class ArbitraryDataCleanupManager extends Thread {
LOGGER.debug(String.format("Transaction %s has complete file and all chunks", LOGGER.debug(String.format("Transaction %s has complete file and all chunks",
Base58.encode(arbitraryTransactionData.getSignature()))); Base58.encode(arbitraryTransactionData.getSignature())));
boolean wasDeleted = ArbitraryTransactionUtils.deleteCompleteFile(arbitraryTransactionData, now, STALE_FILE_TIMEOUT); ArbitraryTransactionUtils.deleteCompleteFile(arbitraryTransactionData, now, STALE_FILE_TIMEOUT);
if( wasDeleted ) {
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"deleting file, retaining chunks",
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
}
continue; continue;
} }
@ -307,6 +236,17 @@ public class ArbitraryDataCleanupManager extends Thread {
this.storageLimitReached(repository); this.storageLimitReached(repository);
} }
// Delete random data associated with name if we're over our storage limit for this name
// Use the DELETION_THRESHOLD, for the same reasons as above
for (String followedName : ListUtils.followedNames()) {
if (isStopping) {
return;
}
if (!storageManager.isStorageSpaceAvailableForName(repository, followedName, DELETION_THRESHOLD)) {
this.storageLimitReachedForName(repository, followedName);
}
}
} catch (DataException e) { } catch (DataException e) {
LOGGER.error("Repository issue when cleaning up arbitrary transaction data", e); LOGGER.error("Repository issue when cleaning up arbitrary transaction data", e);
} }
@ -385,6 +325,25 @@ public class ArbitraryDataCleanupManager extends Thread {
// FUTURE: consider reducing the expiry time of the reader cache // FUTURE: consider reducing the expiry time of the reader cache
} }
public void storageLimitReachedForName(Repository repository, String name) throws InterruptedException {
// We think that the storage limit has been reached for supplied name - but we should double check
if (ArbitraryDataStorageManager.getInstance().isStorageSpaceAvailableForName(repository, name, DELETION_THRESHOLD)) {
// We have space available for this name, so don't delete anything
return;
}
// Delete a batch of random chunks associated with this name
// This reduces the chance of too many nodes deleting the same chunk
// when they reach their storage limit
Path dataPath = Paths.get(Settings.getInstance().getDataPath());
for (int i=0; i<CHUNK_DELETION_BATCH_SIZE; i++) {
if (isStopping) {
return;
}
this.deleteRandomFile(repository, dataPath.toFile(), name);
}
}
/** /**
* Iteratively walk through given directory and delete a single random file * Iteratively walk through given directory and delete a single random file
* *
@ -463,7 +422,6 @@ public class ArbitraryDataCleanupManager extends Thread {
} }
LOGGER.info("Deleting random file {} because we have reached max storage capacity...", randomItem.toString()); LOGGER.info("Deleting random file {} because we have reached max storage capacity...", randomItem.toString());
fireRandomItemDeletionNotification(randomItem, repository, "Deleting random file, because we have reached max storage capacity");
boolean success = randomItem.delete(); boolean success = randomItem.delete();
if (success) { if (success) {
try { try {
@ -478,35 +436,6 @@ public class ArbitraryDataCleanupManager extends Thread {
return false; return false;
} }
private void fireRandomItemDeletionNotification(File randomItem, Repository repository, String reason) {
try {
Path parentFileNamePath = randomItem.toPath().toAbsolutePath().getParent().getFileName();
if (parentFileNamePath != null) {
String signature58 = parentFileNamePath.toString();
byte[] signature = Base58.decode(signature58);
TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
if (transactionData != null && transactionData.getType() == Transaction.TransactionType.ARBITRARY) {
ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
reason,
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
}
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
private void cleanupTempDirectory(String folder, long now, long minAge) { private void cleanupTempDirectory(String folder, long now, long minAge) {
String baseDir = Settings.getInstance().getTempDataPath(); String baseDir = Settings.getInstance().getTempDataPath();
Path tempDir = Paths.get(baseDir, folder); Path tempDir = Paths.get(baseDir, folder);

View File

@ -1,21 +0,0 @@
package org.qortal.controller.arbitrary;
public class ArbitraryDataExamination {
private boolean pass;
private String notes;
public ArbitraryDataExamination(boolean pass, String notes) {
this.pass = pass;
this.notes = notes;
}
public boolean isPass() {
return pass;
}
public String getNotes() {
return notes;
}
}

View File

@ -25,10 +25,6 @@ import org.qortal.utils.NTP;
import org.qortal.utils.Triple; import org.qortal.utils.Triple;
import java.util.*; import java.util.*;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import static org.qortal.controller.arbitrary.ArbitraryDataFileManager.MAX_FILE_HASH_RESPONSES; import static org.qortal.controller.arbitrary.ArbitraryDataFileManager.MAX_FILE_HASH_RESPONSES;
@ -77,8 +73,6 @@ public class ArbitraryDataFileListManager {
private ArbitraryDataFileListManager() { private ArbitraryDataFileListManager() {
getArbitraryDataFileListMessageScheduler.scheduleAtFixedRate(this::processNetworkGetArbitraryDataFileListMessage, 60, 1, TimeUnit.SECONDS);
arbitraryDataFileListMessageScheduler.scheduleAtFixedRate(this::processNetworkArbitraryDataFileListMessage, 60, 1, TimeUnit.SECONDS);
} }
public static ArbitraryDataFileListManager getInstance() { public static ArbitraryDataFileListManager getInstance() {
@ -124,8 +118,8 @@ public class ArbitraryDataFileListManager {
if (timeSinceLastAttempt > 15 * 1000L) { if (timeSinceLastAttempt > 15 * 1000L) {
// We haven't tried for at least 15 seconds // We haven't tried for at least 15 seconds
if (networkBroadcastCount < 12) { if (networkBroadcastCount < 3) {
// We've made less than 12 total attempts // We've made less than 3 total attempts
return true; return true;
} }
} }
@ -134,8 +128,8 @@ public class ArbitraryDataFileListManager {
if (timeSinceLastAttempt > 60 * 1000L) { if (timeSinceLastAttempt > 60 * 1000L) {
// We haven't tried for at least 1 minute // We haven't tried for at least 1 minute
if (networkBroadcastCount < 40) { if (networkBroadcastCount < 8) {
// We've made less than 40 total attempts // We've made less than 8 total attempts
return true; return true;
} }
} }
@ -402,11 +396,11 @@ public class ArbitraryDataFileListManager {
return true; return true;
} }
public void deleteFileListRequestsForSignature(String signature58) { public void deleteFileListRequestsForSignature(byte[] signature) {
String signature58 = Base58.encode(signature);
for (Iterator<Map.Entry<Integer, Triple<String, Peer, Long>>> it = arbitraryDataFileListRequests.entrySet().iterator(); it.hasNext();) { for (Iterator<Map.Entry<Integer, Triple<String, Peer, Long>>> it = arbitraryDataFileListRequests.entrySet().iterator(); it.hasNext();) {
Map.Entry<Integer, Triple<String, Peer, Long>> entry = it.next(); Map.Entry<Integer, Triple<String, Peer, Long>> entry = it.next();
if (entry == null || entry.getKey() == null || entry.getValue() == null) { if (entry == null || entry.getKey() == null || entry.getValue() != null) {
continue; continue;
} }
if (Objects.equals(entry.getValue().getA(), signature58)) { if (Objects.equals(entry.getValue().getA(), signature58)) {
@ -419,116 +413,70 @@ public class ArbitraryDataFileListManager {
// Network handlers // Network handlers
// List to collect messages
private final List<PeerMessage> arbitraryDataFileListMessageList = new ArrayList<>();
// Lock to synchronize access to the list
private final Object arbitraryDataFileListMessageLock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService arbitraryDataFileListMessageScheduler = Executors.newScheduledThreadPool(1);
public void onNetworkArbitraryDataFileListMessage(Peer peer, Message message) { public void onNetworkArbitraryDataFileListMessage(Peer peer, Message message) {
// Don't process if QDN is disabled // Don't process if QDN is disabled
if (!Settings.getInstance().isQdnEnabled()) { if (!Settings.getInstance().isQdnEnabled()) {
return; return;
} }
synchronized (arbitraryDataFileListMessageLock) { ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
arbitraryDataFileListMessageList.add(new PeerMessage(peer, message)); LOGGER.debug("Received hash list from peer {} with {} hashes", peer, arbitraryDataFileListMessage.getHashes().size());
if (LOGGER.isDebugEnabled() && arbitraryDataFileListMessage.getRequestTime() != null) {
long totalRequestTime = NTP.getTime() - arbitraryDataFileListMessage.getRequestTime();
LOGGER.debug("totalRequestTime: {}, requestHops: {}, peerAddress: {}, isRelayPossible: {}",
totalRequestTime, arbitraryDataFileListMessage.getRequestHops(),
arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
} }
}
private void processNetworkArbitraryDataFileListMessage() { // Do we have a pending request for this data?
Triple<String, Peer, Long> request = arbitraryDataFileListRequests.get(message.getId());
if (request == null || request.getA() == null) {
return;
}
boolean isRelayRequest = (request.getB() != null);
try { // Does this message's signature match what we're expecting?
List<PeerMessage> messagesToProcess; byte[] signature = arbitraryDataFileListMessage.getSignature();
synchronized (arbitraryDataFileListMessageLock) { String signature58 = Base58.encode(signature);
messagesToProcess = new ArrayList<>(arbitraryDataFileListMessageList); if (!request.getA().equals(signature58)) {
arbitraryDataFileListMessageList.clear(); return;
} }
if (messagesToProcess.isEmpty()) return; List<byte[]> hashes = arbitraryDataFileListMessage.getHashes();
if (hashes == null || hashes.isEmpty()) {
return;
}
Map<String, PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size()); ArbitraryTransactionData arbitraryTransactionData = null;
Map<String, byte[]> signatureBySignature58 = new HashMap<>(messagesToProcess.size());
Map<String, Boolean> isRelayRequestBySignature58 = new HashMap<>(messagesToProcess.size());
Map<String, List<byte[]>> hashesBySignature58 = new HashMap<>(messagesToProcess.size());
Map<String, Triple<String, Peer, Long>> requestBySignature58 = new HashMap<>(messagesToProcess.size());
for (PeerMessage peerMessage : messagesToProcess) { // Check transaction exists and hashes are correct
Peer peer = peerMessage.getPeer(); try (final Repository repository = RepositoryManager.getRepository()) {
Message message = peerMessage.getMessage(); TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
if (!(transactionData instanceof ArbitraryTransactionData))
ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
LOGGER.debug("Received hash list from peer {} with {} hashes", peer, arbitraryDataFileListMessage.getHashes().size());
if (LOGGER.isDebugEnabled() && arbitraryDataFileListMessage.getRequestTime() != null) {
long totalRequestTime = NTP.getTime() - arbitraryDataFileListMessage.getRequestTime();
LOGGER.debug("totalRequestTime: {}, requestHops: {}, peerAddress: {}, isRelayPossible: {}",
totalRequestTime, arbitraryDataFileListMessage.getRequestHops(),
arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
}
// Do we have a pending request for this data?
Triple<String, Peer, Long> request = arbitraryDataFileListRequests.get(message.getId());
if (request == null || request.getA() == null) {
continue;
}
boolean isRelayRequest = (request.getB() != null);
// Does this message's signature match what we're expecting?
byte[] signature = arbitraryDataFileListMessage.getSignature();
String signature58 = Base58.encode(signature);
if (!request.getA().equals(signature58)) {
continue;
}
List<byte[]> hashes = arbitraryDataFileListMessage.getHashes();
if (hashes == null || hashes.isEmpty()) {
continue;
}
peerMessageBySignature58.put(signature58, peerMessage);
signatureBySignature58.put(signature58, signature);
isRelayRequestBySignature58.put(signature58, isRelayRequest);
hashesBySignature58.put(signature58, hashes);
requestBySignature58.put(signature58, request);
}
if (signatureBySignature58.isEmpty()) return;
List<ArbitraryTransactionData> arbitraryTransactionDataList;
// Check transaction exists and hashes are correct
try (final Repository repository = RepositoryManager.getRepository()) {
arbitraryTransactionDataList
= repository.getTransactionRepository()
.fromSignatures(new ArrayList<>(signatureBySignature58.values())).stream()
.filter(data -> data instanceof ArbitraryTransactionData)
.map(data -> (ArbitraryTransactionData) data)
.collect(Collectors.toList());
} catch (DataException e) {
LOGGER.error(String.format("Repository issue while finding arbitrary transaction data list"), e);
return; return;
}
for (ArbitraryTransactionData arbitraryTransactionData : arbitraryTransactionDataList) { arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
byte[] signature = arbitraryTransactionData.getSignature(); // // Load data file(s)
String signature58 = Base58.encode(signature); // ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
//
// // Check all hashes exist
// for (byte[] hash : hashes) {
// //LOGGER.debug("Received hash {}", Base58.encode(hash));
// if (!arbitraryDataFile.containsChunk(hash)) {
// // Check the hash against the complete file
// if (!Arrays.equals(arbitraryDataFile.getHash(), hash)) {
// LOGGER.info("Received non-matching chunk hash {} for signature {}. This could happen if we haven't obtained the metadata file yet.", Base58.encode(hash), signature58);
// return;
// }
// }
// }
List<byte[]> hashes = hashesBySignature58.get(signature58); if (!isRelayRequest || !Settings.getInstance().isRelayModeEnabled()) {
Long now = NTP.getTime();
PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
Peer peer = peerMessage.getPeer();
Message message = peerMessage.getMessage();
ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
Boolean isRelayRequest = isRelayRequestBySignature58.get(signature58);
if (!isRelayRequest || !Settings.getInstance().isRelayModeEnabled()) {
Long now = NTP.getTime();
if (ArbitraryDataFileManager.getInstance().arbitraryDataFileHashResponses.size() < MAX_FILE_HASH_RESPONSES) {
// Keep track of the hashes this peer reports to have access to // Keep track of the hashes this peer reports to have access to
for (byte[] hash : hashes) { for (byte[] hash : hashes) {
String hash58 = Base58.encode(hash); String hash58 = Base58.encode(hash);
@ -539,300 +487,233 @@ public class ArbitraryDataFileListManager {
ArbitraryFileListResponseInfo responseInfo = new ArbitraryFileListResponseInfo(hash58, signature58, ArbitraryFileListResponseInfo responseInfo = new ArbitraryFileListResponseInfo(hash58, signature58,
peer, now, arbitraryDataFileListMessage.getRequestTime(), requestHops); peer, now, arbitraryDataFileListMessage.getRequestTime(), requestHops);
ArbitraryDataFileManager.getInstance().addResponse(responseInfo); ArbitraryDataFileManager.getInstance().arbitraryDataFileHashResponses.add(responseInfo);
}
// Keep track of the source peer, for direct connections
if (arbitraryDataFileListMessage.getPeerAddress() != null) {
ArbitraryDataFileManager.getInstance().addDirectConnectionInfoIfUnique(
new ArbitraryDirectConnectionInfo(signature, arbitraryDataFileListMessage.getPeerAddress(), hashes, now));
} }
} }
// Forwarding // Keep track of the source peer, for direct connections
if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) { if (arbitraryDataFileListMessage.getPeerAddress() != null) {
ArbitraryDataFileManager.getInstance().addDirectConnectionInfoIfUnique(
new ArbitraryDirectConnectionInfo(signature, arbitraryDataFileListMessage.getPeerAddress(), hashes, now));
}
}
boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName())); } catch (DataException e) {
if (!isBlocked) { LOGGER.error(String.format("Repository issue while finding arbitrary transaction data list for peer %s", peer), e);
Triple<String, Peer, Long> request = requestBySignature58.get(signature58); }
Peer requestingPeer = request.getB();
if (requestingPeer != null) {
Long requestTime = arbitraryDataFileListMessage.getRequestTime();
Integer requestHops = arbitraryDataFileListMessage.getRequestHops();
// Add each hash to our local mapping so we know who to ask later // Forwarding
Long now = NTP.getTime(); if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
for (byte[] hash : hashes) { boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
String hash58 = Base58.encode(hash); if (!isBlocked) {
ArbitraryRelayInfo relayInfo = new ArbitraryRelayInfo(hash58, signature58, peer, now, requestTime, requestHops); Peer requestingPeer = request.getB();
ArbitraryDataFileManager.getInstance().addToRelayMap(relayInfo); if (requestingPeer != null) {
} Long requestTime = arbitraryDataFileListMessage.getRequestTime();
Integer requestHops = arbitraryDataFileListMessage.getRequestHops();
// Bump requestHops if it exists // Add each hash to our local mapping so we know who to ask later
if (requestHops != null) { Long now = NTP.getTime();
requestHops++; for (byte[] hash : hashes) {
} String hash58 = Base58.encode(hash);
ArbitraryRelayInfo relayInfo = new ArbitraryRelayInfo(hash58, signature58, peer, now, requestTime, requestHops);
ArbitraryDataFileManager.getInstance().addToRelayMap(relayInfo);
}
ArbitraryDataFileListMessage forwardArbitraryDataFileListMessage; // Bump requestHops if it exists
if (requestHops != null) {
requestHops++;
}
// Remove optional parameters if the requesting peer doesn't support it yet ArbitraryDataFileListMessage forwardArbitraryDataFileListMessage;
// A message with less statistical data is better than no message at all
if (!requestingPeer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
} else {
forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops,
arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
}
forwardArbitraryDataFileListMessage.setId(message.getId());
// Forward to requesting peer // Remove optional parameters if the requesting peer doesn't support it yet
LOGGER.debug("Forwarding file list with {} hashes to requesting peer: {}", hashes.size(), requestingPeer); // A message with less statistical data is better than no message at all
requestingPeer.sendMessage(forwardArbitraryDataFileListMessage); if (!requestingPeer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
} forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
} else {
forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops,
arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
}
forwardArbitraryDataFileListMessage.setId(message.getId());
// Forward to requesting peer
LOGGER.debug("Forwarding file list with {} hashes to requesting peer: {}", hashes.size(), requestingPeer);
if (!requestingPeer.sendMessage(forwardArbitraryDataFileListMessage)) {
requestingPeer.disconnect("failed to forward arbitrary data file list");
} }
} }
} }
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
} }
} }
// List to collect messages
private final List<PeerMessage> getArbitraryDataFileListMessageList = new ArrayList<>();
// Lock to synchronize access to the list
private final Object getArbitraryDataFileListMessageLock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService getArbitraryDataFileListMessageScheduler = Executors.newScheduledThreadPool(1);
public void onNetworkGetArbitraryDataFileListMessage(Peer peer, Message message) { public void onNetworkGetArbitraryDataFileListMessage(Peer peer, Message message) {
// Don't respond if QDN is disabled // Don't respond if QDN is disabled
if (!Settings.getInstance().isQdnEnabled()) { if (!Settings.getInstance().isQdnEnabled()) {
return; return;
} }
synchronized (getArbitraryDataFileListMessageLock) { Controller.getInstance().stats.getArbitraryDataFileListMessageStats.requests.incrementAndGet();
getArbitraryDataFileListMessageList.add(new PeerMessage(peer, message));
GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
byte[] signature = getArbitraryDataFileListMessage.getSignature();
String signature58 = Base58.encode(signature);
Long now = NTP.getTime();
Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peer, now);
// If we've seen this request recently, then ignore
if (arbitraryDataFileListRequests.putIfAbsent(message.getId(), newEntry) != null) {
LOGGER.trace("Ignoring hash list request from peer {} for signature {}", peer, signature58);
return;
} }
}
private void processNetworkGetArbitraryDataFileListMessage() { List<byte[]> requestedHashes = getArbitraryDataFileListMessage.getHashes();
int hashCount = requestedHashes != null ? requestedHashes.size() : 0;
String requestingPeer = getArbitraryDataFileListMessage.getRequestingPeer();
try { if (requestingPeer != null) {
List<PeerMessage> messagesToProcess; LOGGER.debug("Received hash list request with {} hashes from peer {} (requesting peer {}) for signature {}", hashCount, peer, requestingPeer, signature58);
synchronized (getArbitraryDataFileListMessageLock) { }
messagesToProcess = new ArrayList<>(getArbitraryDataFileListMessageList); else {
getArbitraryDataFileListMessageList.clear(); LOGGER.debug("Received hash list request with {} hashes from peer {} for signature {}", hashCount, peer, signature58);
} }
if (messagesToProcess.isEmpty()) return; List<byte[]> hashes = new ArrayList<>();
ArbitraryTransactionData transactionData = null;
boolean allChunksExist = false;
boolean hasMetadata = false;
Map<String, byte[]> signatureBySignature58 = new HashMap<>(messagesToProcess.size()); try (final Repository repository = RepositoryManager.getRepository()) {
Map<String, List<byte[]>> requestedHashesBySignature58 = new HashMap<>(messagesToProcess.size());
Map<String, String> requestingPeerBySignature58 = new HashMap<>(messagesToProcess.size());
Map<String, Long> nowBySignature58 = new HashMap<>((messagesToProcess.size()));
Map<String, PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
for (PeerMessage messagePeer : messagesToProcess) { // Firstly we need to lookup this file on chain to get a list of its hashes
Controller.getInstance().stats.getArbitraryDataFileListMessageStats.requests.incrementAndGet(); transactionData = (ArbitraryTransactionData)repository.getTransactionRepository().fromSignature(signature);
if (transactionData instanceof ArbitraryTransactionData) {
Message message = messagePeer.message;
Peer peer = messagePeer.peer;
GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
byte[] signature = getArbitraryDataFileListMessage.getSignature();
String signature58 = Base58.encode(signature);
Long now = NTP.getTime();
Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peer, now);
// If we've seen this request recently, then ignore
if (arbitraryDataFileListRequests.putIfAbsent(message.getId(), newEntry) != null) {
LOGGER.trace("Ignoring hash list request from peer {} for signature {}", peer, signature58);
continue;
}
List<byte[]> requestedHashes = getArbitraryDataFileListMessage.getHashes();
int hashCount = requestedHashes != null ? requestedHashes.size() : 0;
String requestingPeer = getArbitraryDataFileListMessage.getRequestingPeer();
if (requestingPeer != null) {
LOGGER.debug("Received hash list request with {} hashes from peer {} (requesting peer {}) for signature {}", hashCount, peer, requestingPeer, signature58);
} else {
LOGGER.debug("Received hash list request with {} hashes from peer {} for signature {}", hashCount, peer, signature58);
}
signatureBySignature58.put(signature58, signature);
requestedHashesBySignature58.put(signature58, requestedHashes);
requestingPeerBySignature58.put(signature58, requestingPeer);
nowBySignature58.put(signature58, now);
peerMessageBySignature58.put(signature58, messagePeer);
}
if (signatureBySignature58.isEmpty()) {
return;
}
List<byte[]> hashes = new ArrayList<>();
boolean allChunksExist = false;
boolean hasMetadata = false;
List<ArbitraryTransactionData> transactionDataList;
try (final Repository repository = RepositoryManager.getRepository()) {
// Firstly we need to lookup this file on chain to get a list of its hashes
transactionDataList
= repository.getTransactionRepository()
.fromSignatures(new ArrayList<>(signatureBySignature58.values())).stream()
.filter(data -> data instanceof ArbitraryTransactionData)
.map(data -> (ArbitraryTransactionData) data)
.collect(Collectors.toList());
} catch (DataException e) {
LOGGER.error(String.format("Repository issue while fetching arbitrary file list for peer"), e);
return;
}
for (ArbitraryTransactionData transactionData : transactionDataList) {
byte[] signature = transactionData.getSignature();
String signature58 = Base58.encode(signature);
List<byte[]> requestedHashes = requestedHashesBySignature58.get(signature58);
// Check if we're even allowed to serve data for this transaction // Check if we're even allowed to serve data for this transaction
if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) { if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
try { // Load file(s) and add any that exist to the list of hashes
// Load file(s) and add any that exist to the list of hashes ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
// If the peer didn't supply a hash list, we need to return all hashes for this transaction // If the peer didn't supply a hash list, we need to return all hashes for this transaction
if (requestedHashes == null || requestedHashes.isEmpty()) { if (requestedHashes == null || requestedHashes.isEmpty()) {
requestedHashes = new ArrayList<>(); requestedHashes = new ArrayList<>();
// Add the metadata file // Add the metadata file
if (arbitraryDataFile.getMetadataHash() != null) { if (arbitraryDataFile.getMetadataHash() != null) {
requestedHashes.add(arbitraryDataFile.getMetadataHash()); requestedHashes.add(arbitraryDataFile.getMetadataHash());
hasMetadata = true; hasMetadata = true;
}
// Add the chunk hashes
if (!arbitraryDataFile.getChunkHashes().isEmpty()) {
requestedHashes.addAll(arbitraryDataFile.getChunkHashes());
}
// Add complete file if there are no hashes
else {
requestedHashes.add(arbitraryDataFile.getHash());
}
} }
// Add the chunk hashes
// Assume all chunks exists, unless one can't be found below if (!arbitraryDataFile.getChunkHashes().isEmpty()) {
allChunksExist = true; requestedHashes.addAll(arbitraryDataFile.getChunkHashes());
}
for (byte[] requestedHash : requestedHashes) { // Add complete file if there are no hashes
ArbitraryDataFileChunk chunk = ArbitraryDataFileChunk.fromHash(requestedHash, signature); else {
if (chunk.exists()) { requestedHashes.add(arbitraryDataFile.getHash());
hashes.add(chunk.getHash());
//LOGGER.trace("Added hash {}", chunk.getHash58());
} else {
LOGGER.trace("Couldn't add hash {} because it doesn't exist", chunk.getHash58());
allChunksExist = false;
}
} }
} catch (DataException e) {
LOGGER.error(e.getMessage(), e);
}
}
// If the only file we have is the metadata then we shouldn't respond. Most nodes will already have that,
// or can use the separate metadata protocol to fetch it. This should greatly reduce network spam.
if (hasMetadata && hashes.size() == 1) {
hashes.clear();
}
PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
Peer peer = peerMessage.getPeer();
Message message = peerMessage.getMessage();
Long now = nowBySignature58.get(signature58);
// We should only respond if we have at least one hash
String requestingPeer = requestingPeerBySignature58.get(signature58);
if (!hashes.isEmpty()) {
// Firstly we should keep track of the requesting peer, to allow for potential direct connections later
ArbitraryDataFileManager.getInstance().addRecentDataRequest(requestingPeer);
// We have all the chunks, so update requests map to reflect that we've sent it
// There is no need to keep track of the request, as we can serve all the chunks
if (allChunksExist) {
Triple<String, Peer, Long> newEntry = new Triple<>(null, null, now);
arbitraryDataFileListRequests.put(message.getId(), newEntry);
} }
String ourAddress = Network.getInstance().getOurExternalIpAddressAndPort(); // Assume all chunks exists, unless one can't be found below
ArbitraryDataFileListMessage arbitraryDataFileListMessage; allChunksExist = true;
// Remove optional parameters if the requesting peer doesn't support it yet
// A message with less statistical data is better than no message at all
if (!peer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
} else {
arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature,
hashes, NTP.getTime(), 0, ourAddress, true);
}
arbitraryDataFileListMessage.setId(message.getId());
if (!peer.sendMessage(arbitraryDataFileListMessage)) {
LOGGER.debug("Couldn't send list of hashes");
continue;
}
if (allChunksExist) {
// Nothing left to do, so return to prevent any unnecessary forwarding from occurring
LOGGER.debug("No need for any forwarding because file list request is fully served");
continue;
}
}
// We may need to forward this request on
boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
// In relay mode - so ask our other peers if they have it
GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
long requestTime = getArbitraryDataFileListMessage.getRequestTime();
int requestHops = getArbitraryDataFileListMessage.getRequestHops() + 1;
long totalRequestTime = now - requestTime;
if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
// Relay request hasn't timed out yet, so can potentially be rebroadcast
if (requestHops < RELAY_REQUEST_MAX_HOPS) {
// Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
Message relayGetArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops, requestingPeer);
relayGetArbitraryDataFileListMessage.setId(message.getId());
LOGGER.debug("Rebroadcasting hash list request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
Network.getInstance().broadcast(
broadcastPeer ->
!broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryDataFileListMessage
);
for (byte[] requestedHash : requestedHashes) {
ArbitraryDataFileChunk chunk = ArbitraryDataFileChunk.fromHash(requestedHash, signature);
if (chunk.exists()) {
hashes.add(chunk.getHash());
//LOGGER.trace("Added hash {}", chunk.getHash58());
} else { } else {
// This relay request has reached the maximum number of allowed hops LOGGER.trace("Couldn't add hash {} because it doesn't exist", chunk.getHash58());
allChunksExist = false;
} }
} else {
// This relay request has timed out
} }
} }
} }
} catch (Exception e) {
LOGGER.error(e.getMessage(), e); } catch (DataException e) {
LOGGER.error(String.format("Repository issue while fetching arbitrary file list for peer %s", peer), e);
}
// If the only file we have is the metadata then we shouldn't respond. Most nodes will already have that,
// or can use the separate metadata protocol to fetch it. This should greatly reduce network spam.
if (hasMetadata && hashes.size() == 1) {
hashes.clear();
}
// We should only respond if we have at least one hash
if (!hashes.isEmpty()) {
// Firstly we should keep track of the requesting peer, to allow for potential direct connections later
ArbitraryDataFileManager.getInstance().addRecentDataRequest(requestingPeer);
// We have all the chunks, so update requests map to reflect that we've sent it
// There is no need to keep track of the request, as we can serve all the chunks
if (allChunksExist) {
newEntry = new Triple<>(null, null, now);
arbitraryDataFileListRequests.put(message.getId(), newEntry);
}
String ourAddress = Network.getInstance().getOurExternalIpAddressAndPort();
ArbitraryDataFileListMessage arbitraryDataFileListMessage;
// Remove optional parameters if the requesting peer doesn't support it yet
// A message with less statistical data is better than no message at all
if (!peer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
} else {
arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature,
hashes, NTP.getTime(), 0, ourAddress, true);
}
arbitraryDataFileListMessage.setId(message.getId());
if (!peer.sendMessage(arbitraryDataFileListMessage)) {
LOGGER.debug("Couldn't send list of hashes");
peer.disconnect("failed to send list of hashes");
return;
}
LOGGER.debug("Sent list of hashes (count: {})", hashes.size());
if (allChunksExist) {
// Nothing left to do, so return to prevent any unnecessary forwarding from occurring
LOGGER.debug("No need for any forwarding because file list request is fully served");
return;
}
}
// We may need to forward this request on
boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
// In relay mode - so ask our other peers if they have it
long requestTime = getArbitraryDataFileListMessage.getRequestTime();
int requestHops = getArbitraryDataFileListMessage.getRequestHops() + 1;
long totalRequestTime = now - requestTime;
if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
// Relay request hasn't timed out yet, so can potentially be rebroadcast
if (requestHops < RELAY_REQUEST_MAX_HOPS) {
// Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
Message relayGetArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops, requestingPeer);
relayGetArbitraryDataFileListMessage.setId(message.getId());
LOGGER.debug("Rebroadcasting hash list request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
Network.getInstance().broadcast(
broadcastPeer ->
!broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryDataFileListMessage
);
}
else {
// This relay request has reached the maximum number of allowed hops
}
}
else {
// This relay request has timed out
}
} }
} }

View File

@ -1,7 +1,6 @@
package org.qortal.controller.arbitrary; package org.qortal.controller.arbitrary;
import com.google.common.net.InetAddresses; import com.google.common.net.InetAddresses;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.qortal.arbitrary.ArbitraryDataFile; import org.qortal.arbitrary.ArbitraryDataFile;
@ -24,15 +23,10 @@ import org.qortal.utils.NTP;
import java.security.SecureRandom; import java.security.SecureRandom;
import java.util.*; import java.util.*;
import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutorService;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.Executors; import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.qortal.network.PeerSendManager;
public class ArbitraryDataFileManager extends Thread { public class ArbitraryDataFileManager extends Thread {
private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFileManager.class); private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFileManager.class);
@ -54,7 +48,7 @@ public class ArbitraryDataFileManager extends Thread {
/** /**
* List to keep track of any arbitrary data file hash responses * List to keep track of any arbitrary data file hash responses
*/ */
private final List<ArbitraryFileListResponseInfo> arbitraryDataFileHashResponses = Collections.synchronizedList(new ArrayList<>()); public final List<ArbitraryFileListResponseInfo> arbitraryDataFileHashResponses = Collections.synchronizedList(new ArrayList<>());
/** /**
* List to keep track of peers potentially available for direct connections, based on recent requests * List to keep track of peers potentially available for direct connections, based on recent requests
@ -70,38 +64,9 @@ public class ArbitraryDataFileManager extends Thread {
public static int MAX_FILE_HASH_RESPONSES = 1000; public static int MAX_FILE_HASH_RESPONSES = 1000;
private final Map<Peer, PeerSendManager> peerSendManagers = new ConcurrentHashMap<>();
private PeerSendManager getOrCreateSendManager(Peer peer) {
return peerSendManagers.computeIfAbsent(peer, p -> new PeerSendManager(p));
}
private ArbitraryDataFileManager() { private ArbitraryDataFileManager() {
this.arbitraryDataFileHashResponseScheduler.scheduleAtFixedRate( this::processResponses, 60, 1, TimeUnit.SECONDS);
this.arbitraryDataFileHashResponseScheduler.scheduleAtFixedRate(this::handleFileListRequestProcess, 60, 1, TimeUnit.SECONDS);
ScheduledExecutorService cleaner = Executors.newSingleThreadScheduledExecutor();
cleaner.scheduleAtFixedRate(() -> {
long idleCutoff = TimeUnit.MINUTES.toMillis(2);
Iterator<Map.Entry<Peer, PeerSendManager>> iterator = peerSendManagers.entrySet().iterator();
while (iterator.hasNext()) {
Map.Entry<Peer, PeerSendManager> entry = iterator.next();
Peer peer = entry.getKey();
PeerSendManager manager = entry.getValue();
if (manager.isIdle(idleCutoff)) {
iterator.remove(); // SAFE removal during iteration
manager.shutdown();
LOGGER.debug("Cleaned up PeerSendManager for peer {}", peer);
}
}
}, 0, 5, TimeUnit.MINUTES);
} }
public static ArbitraryDataFileManager getInstance() { public static ArbitraryDataFileManager getInstance() {
@ -111,13 +76,18 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
return instance; return instance;
} }
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data File Manager"); Thread.currentThread().setName("Arbitrary Data File Manager");
try { try {
// Use a fixed thread pool to execute the arbitrary data file requests
int threadCount = 5;
ExecutorService arbitraryDataFileRequestExecutor = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++) {
arbitraryDataFileRequestExecutor.execute(new ArbitraryDataFileRequestThread());
}
while (!isStopping) { while (!isStopping) {
// Nothing to do yet // Nothing to do yet
Thread.sleep(1000); Thread.sleep(1000);
@ -142,6 +112,7 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
final long relayMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_RELAY_TIMEOUT; final long relayMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_RELAY_TIMEOUT;
arbitraryRelayMap.removeIf(entry -> entry == null || entry.getTimestamp() == null || entry.getTimestamp() < relayMinimumTimestamp); arbitraryRelayMap.removeIf(entry -> entry == null || entry.getTimestamp() == null || entry.getTimestamp() < relayMinimumTimestamp);
arbitraryDataFileHashResponses.removeIf(entry -> entry.getTimestamp() < relayMinimumTimestamp);
final long directConnectionInfoMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT; final long directConnectionInfoMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT;
directConnectionInfo.removeIf(entry -> entry.getTimestamp() < directConnectionInfoMinimumTimestamp); directConnectionInfo.removeIf(entry -> entry.getTimestamp() < directConnectionInfoMinimumTimestamp);
@ -154,7 +125,8 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
// Fetch data files by hash // Fetch data files by hash
public boolean fetchArbitraryDataFiles(Peer peer, public boolean fetchArbitraryDataFiles(Repository repository,
Peer peer,
byte[] signature, byte[] signature,
ArbitraryTransactionData arbitraryTransactionData, ArbitraryTransactionData arbitraryTransactionData,
List<byte[]> hashes) throws DataException { List<byte[]> hashes) throws DataException {
@ -174,15 +146,21 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) { if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) {
LOGGER.debug("Requesting data file {} from peer {}", hash58, peer); LOGGER.debug("Requesting data file {} from peer {}", hash58, peer);
Long startTime = NTP.getTime(); Long startTime = NTP.getTime();
ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, arbitraryTransactionData, signature, hash); ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, null, arbitraryTransactionData, signature, hash, null);
Long endTime = NTP.getTime(); Long endTime = NTP.getTime();
if (receivedArbitraryDataFile != null) { if (receivedArbitraryDataFile != null) {
LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime)); LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime));
receivedAtLeastOneFile = true; receivedAtLeastOneFile = true;
// Remove this hash from arbitraryDataFileHashResponses now that we have received it
arbitraryDataFileHashResponses.remove(hash58);
} }
else { else {
LOGGER.debug("Peer {} didn't respond with data file {} for signature {}. Time taken: {} ms", peer, Base58.encode(hash), Base58.encode(signature), (endTime-startTime)); LOGGER.debug("Peer {} didn't respond with data file {} for signature {}. Time taken: {} ms", peer, Base58.encode(hash), Base58.encode(signature), (endTime-startTime));
// Remove this hash from arbitraryDataFileHashResponses now that we have failed to receive it
arbitraryDataFileHashResponses.remove(hash58);
// Stop asking for files from this peer // Stop asking for files from this peer
break; break;
} }
@ -191,6 +169,10 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
LOGGER.trace("Already requesting data file {} for signature {} from peer {}", arbitraryDataFile, Base58.encode(signature), peer); LOGGER.trace("Already requesting data file {} for signature {} from peer {}", arbitraryDataFile, Base58.encode(signature), peer);
} }
} }
else {
// Remove this hash from arbitraryDataFileHashResponses because we have a local copy
arbitraryDataFileHashResponses.remove(hash58);
}
} }
if (receivedAtLeastOneFile) { if (receivedAtLeastOneFile) {
@ -209,103 +191,14 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
return receivedAtLeastOneFile; return receivedAtLeastOneFile;
} }
// Lock to synchronize access to the list private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, Peer requestingPeer, ArbitraryTransactionData arbitraryTransactionData, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
private final Object arbitraryDataFileHashResponseLock = new Object(); ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature);
boolean fileAlreadyExists = existingFile.exists();
// Scheduled executor service to process messages every second String hash58 = Base58.encode(hash);
private final ScheduledExecutorService arbitraryDataFileHashResponseScheduler = Executors.newScheduledThreadPool(1);
public void addResponse( ArbitraryFileListResponseInfo responseInfo ) {
synchronized (arbitraryDataFileHashResponseLock) {
this.arbitraryDataFileHashResponses.add(responseInfo);
}
}
private void processResponses() {
try {
List<ArbitraryFileListResponseInfo> responsesToProcess;
synchronized (arbitraryDataFileHashResponseLock) {
responsesToProcess = new ArrayList<>(arbitraryDataFileHashResponses);
arbitraryDataFileHashResponses.clear();
}
if (responsesToProcess.isEmpty()) return;
Long now = NTP.getTime();
ArbitraryDataFileRequestThread.getInstance().processFileHashes(now, responsesToProcess, this);
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, ArbitraryTransactionData arbitraryTransactionData, byte[] signature, byte[] hash) throws DataException {
ArbitraryDataFile arbitraryDataFile; ArbitraryDataFile arbitraryDataFile;
try { // Fetch the file if it doesn't exist locally
ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature); if (!fileAlreadyExists) {
boolean fileAlreadyExists = existingFile.exists();
String hash58 = Base58.encode(hash);
// Fetch the file if it doesn't exist locally
if (!fileAlreadyExists) {
LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer));
arbitraryDataFileRequests.put(hash58, NTP.getTime());
Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash);
Message response = null;
try {
response = peer.getResponseWithTimeout(getArbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT);
} catch (InterruptedException e) {
// Will return below due to null response
}
arbitraryDataFileRequests.remove(hash58);
LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58));
if (response == null) {
LOGGER.debug("Received null response from peer {}", peer);
return null;
}
if (response.getType() != MessageType.ARBITRARY_DATA_FILE) {
LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer);
return null;
}
ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
} else {
LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
arbitraryDataFile = existingFile;
}
if (arbitraryDataFile != null) {
arbitraryDataFile.save();
// If this is a metadata file then we need to update the cache
if (arbitraryTransactionData != null && arbitraryTransactionData.getMetadataHash() != null) {
if (Arrays.equals(arbitraryTransactionData.getMetadataHash(), hash)) {
ArbitraryDataCacheManager.getInstance().addToUpdateQueue(arbitraryTransactionData);
}
}
// We may need to remove the file list request, if we have all the files for this transaction
this.handleFileListRequests(signature);
}
} catch (DataException e) {
LOGGER.error(e.getMessage(), e);
arbitraryDataFile = null;
}
return arbitraryDataFile;
}
private void fetchFileForRelay(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
try {
String hash58 = Base58.encode(hash);
LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer)); LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer));
arbitraryDataFileRequests.put(hash58, NTP.getTime()); arbitraryDataFileRequests.put(hash58, NTP.getTime());
Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash); Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash);
@ -319,73 +212,73 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
arbitraryDataFileRequests.remove(hash58); arbitraryDataFileRequests.remove(hash58);
LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58)); LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58));
// We may need to remove the file list request, if we have all the files for this transaction
this.handleFileListRequests(signature);
if (response == null) { if (response == null) {
LOGGER.debug("Received null response from peer {}", peer); LOGGER.debug("Received null response from peer {}", peer);
return; return null;
} }
if (response.getType() != MessageType.ARBITRARY_DATA_FILE) { if (response.getType() != MessageType.ARBITRARY_DATA_FILE) {
LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer); LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer);
return; return null;
} }
ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response; ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
ArbitraryDataFile arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile(); arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
} else {
if (arbitraryDataFile != null) { LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
arbitraryDataFile = existingFile;
// We might want to forward the request to the peer that originally requested it
this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
} }
if (arbitraryDataFile == null) {
// We don't have a file, so give up here
return null;
}
// We might want to forward the request to the peer that originally requested it
this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
boolean isRelayRequest = (requestingPeer != null);
if (isRelayRequest) {
if (!fileAlreadyExists) {
// File didn't exist locally before the request, and it's a forwarding request, so delete it if it exists.
// It shouldn't exist on the filesystem yet, but leaving this here just in case.
arbitraryDataFile.delete(10);
}
}
else {
arbitraryDataFile.save();
}
// If this is a metadata file then we need to update the cache
if (arbitraryTransactionData != null && arbitraryTransactionData.getMetadataHash() != null) {
if (Arrays.equals(arbitraryTransactionData.getMetadataHash(), hash)) {
ArbitraryDataCacheManager.getInstance().addToUpdateQueue(arbitraryTransactionData);
}
}
return arbitraryDataFile;
} }
Map<String, byte[]> signatureBySignature58 = new HashMap<>();
// Lock to synchronize access to the list
private final Object handleFileListRequestsLock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService handleFileListRequestsScheduler = Executors.newScheduledThreadPool(1);
private void handleFileListRequests(byte[] signature) { private void handleFileListRequests(byte[] signature) {
synchronized (handleFileListRequestsLock) {
signatureBySignature58.put(Base58.encode(signature), signature);
}
}
private void handleFileListRequestProcess() {
Map<String, byte[]> signaturesToProcess;
synchronized (handleFileListRequestsLock) {
signaturesToProcess = new HashMap<>(signatureBySignature58);
signatureBySignature58.clear();
}
if( signaturesToProcess.isEmpty() ) return;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
// Fetch the transaction data // Fetch the transaction data
List<ArbitraryTransactionData> arbitraryTransactionDataList ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
= ArbitraryTransactionUtils.fetchTransactionDataList(repository, new ArrayList<>(signaturesToProcess.values())); if (arbitraryTransactionData == null) {
return;
for( ArbitraryTransactionData arbitraryTransactionData : arbitraryTransactionDataList ) {
boolean completeFileExists = ArbitraryTransactionUtils.completeFileExists(arbitraryTransactionData);
if (completeFileExists) {
String signature58 = Base58.encode(arbitraryTransactionData.getSignature());
LOGGER.debug("All chunks or complete file exist for transaction {}", signature58);
ArbitraryDataFileListManager.getInstance().deleteFileListRequestsForSignature(signature58);
}
} }
} catch (Exception e) { boolean allChunksExist = ArbitraryTransactionUtils.allChunksExist(arbitraryTransactionData);
LOGGER.error(e.getMessage(), e);
if (allChunksExist) {
// Update requests map to reflect that we've received all chunks
ArbitraryDataFileListManager.getInstance().deleteFileListRequestsForSignature(signature);
}
} catch (DataException e) {
LOGGER.debug("Unable to handle file list requests: {}", e.getMessage());
} }
} }
@ -402,14 +295,15 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
LOGGER.debug("Received arbitrary data file - forwarding is needed"); LOGGER.debug("Received arbitrary data file - forwarding is needed");
try { // The ID needs to match that of the original request
// The ID needs to match that of the original request message.setId(originalMessage.getId());
message.setId(originalMessage.getId());
getOrCreateSendManager(requestingPeer).queueMessage(message); if (!requestingPeer.sendMessageWithTimeout(message, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
LOGGER.debug("Failed to forward arbitrary data file to peer {}", requestingPeer);
} catch (Exception e) { requestingPeer.disconnect("failed to forward arbitrary data file");
LOGGER.error(e.getMessage(), e); }
else {
LOGGER.debug("Forwarded arbitrary data file to peer {}", requestingPeer);
} }
} }
@ -683,9 +577,13 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
LOGGER.debug("Sending file {}...", arbitraryDataFile); LOGGER.debug("Sending file {}...", arbitraryDataFile);
ArbitraryDataFileMessage arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, arbitraryDataFile); ArbitraryDataFileMessage arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, arbitraryDataFile);
arbitraryDataFileMessage.setId(message.getId()); arbitraryDataFileMessage.setId(message.getId());
if (!peer.sendMessageWithTimeout(arbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
getOrCreateSendManager(peer).queueMessage(arbitraryDataFileMessage); LOGGER.debug("Couldn't send file {}", arbitraryDataFile);
peer.disconnect("failed to send file");
}
else {
LOGGER.debug("Sent file {}", arbitraryDataFile);
}
} }
else if (relayInfo != null) { else if (relayInfo != null) {
LOGGER.debug("We have relay info for hash {}", Base58.encode(hash)); LOGGER.debug("We have relay info for hash {}", Base58.encode(hash));
@ -697,7 +595,7 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
LOGGER.debug("Asking peer {} for hash {}", peerToAsk, hash58); LOGGER.debug("Asking peer {} for hash {}", peerToAsk, hash58);
// No need to pass arbitraryTransactionData below because this is only used for metadata caching, // No need to pass arbitraryTransactionData below because this is only used for metadata caching,
// and metadata isn't retained when relaying. // and metadata isn't retained when relaying.
this.fetchFileForRelay(peerToAsk, peer, signature, hash, message); this.fetchArbitraryDataFile(peerToAsk, peer, null, signature, hash, message);
} }
else { else {
LOGGER.debug("Peer {} not found in relay info", peer); LOGGER.debug("Peer {} not found in relay info", peer);
@ -719,6 +617,7 @@ private PeerSendManager getOrCreateSendManager(Peer peer) {
fileUnknownMessage.setId(message.getId()); fileUnknownMessage.setId(message.getId());
if (!peer.sendMessage(fileUnknownMessage)) { if (!peer.sendMessage(fileUnknownMessage)) {
LOGGER.debug("Couldn't sent file-unknown response"); LOGGER.debug("Couldn't sent file-unknown response");
peer.disconnect("failed to send file-unknown response");
} }
else { else {
LOGGER.debug("Sent file-unknown response for file {}", arbitraryDataFile); LOGGER.debug("Sent file-unknown response for file {}", arbitraryDataFile);

View File

@ -4,186 +4,122 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import org.qortal.controller.Controller; import org.qortal.controller.Controller;
import org.qortal.data.arbitrary.ArbitraryFileListResponseInfo; import org.qortal.data.arbitrary.ArbitraryFileListResponseInfo;
import org.qortal.data.arbitrary.ArbitraryResourceData;
import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.network.Peer; import org.qortal.network.Peer;
import org.qortal.network.message.MessageType;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
import org.qortal.repository.Repository; import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager; import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.utils.ArbitraryTransactionUtils; import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.Base58; import org.qortal.utils.Base58;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import org.qortal.utils.NamedThreadFactory;
import java.net.http.HttpResponse;
import java.util.ArrayList;
import java.util.Arrays; import java.util.Arrays;
import java.util.Comparator; import java.util.Comparator;
import java.util.HashMap;
import java.util.Iterator; import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import static java.lang.Thread.NORM_PRIORITY; public class ArbitraryDataFileRequestThread implements Runnable {
public class ArbitraryDataFileRequestThread {
private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFileRequestThread.class); private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFileRequestThread.class);
private static final Integer FETCHER_LIMIT_PER_PEER = Settings.getInstance().getMaxThreadsForMessageType(MessageType.GET_ARBITRARY_DATA_FILE); public ArbitraryDataFileRequestThread() {
private static final String FETCHER_THREAD_PREFIX = "Arbitrary Data Fetcher ";
private ConcurrentHashMap<String, ExecutorService> executorByPeer = new ConcurrentHashMap<>();
private ArbitraryDataFileRequestThread() {
cleanupExecutorByPeerScheduler.scheduleAtFixedRate(this::cleanupExecutorsByPeer, 1, 1, TimeUnit.MINUTES);
} }
private static ArbitraryDataFileRequestThread instance = null; @Override
public void run() {
public static ArbitraryDataFileRequestThread getInstance() { Thread.currentThread().setName("Arbitrary Data File Request Thread");
if( instance == null ) {
instance = new ArbitraryDataFileRequestThread();
}
return instance;
}
private final ScheduledExecutorService cleanupExecutorByPeerScheduler = Executors.newScheduledThreadPool(1);
private void cleanupExecutorsByPeer() {
try { try {
this.executorByPeer.forEach((key, value) -> { while (!Controller.isStopping()) {
if (value instanceof ThreadPoolExecutor) { Long now = NTP.getTime();
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) value; this.processFileHashes(now);
if (threadPoolExecutor.getActiveCount() == 0) { }
threadPoolExecutor.shutdown(); } catch (InterruptedException e) {
if (this.executorByPeer.computeIfPresent(key, (k, v) -> null) == null) { // Fall-through to exit thread...
LOGGER.trace("removed executor: peer = " + key);
}
}
} else {
LOGGER.warn("casting issue in cleanup");
}
});
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
} }
} }
public void processFileHashes(Long now, List<ArbitraryFileListResponseInfo> responseInfos, ArbitraryDataFileManager arbitraryDataFileManager) { private void processFileHashes(Long now) throws InterruptedException {
if (Controller.isStopping()) { if (Controller.isStopping()) {
return; return;
} }
Map<String, byte[]> signatureBySignature58 = new HashMap<>(responseInfos.size()); ArbitraryDataFileManager arbitraryDataFileManager = ArbitraryDataFileManager.getInstance();
Map<String, List<ArbitraryFileListResponseInfo>> responseInfoBySignature58 = new HashMap<>(); String signature58 = null;
String hash58 = null;
Peer peer = null;
boolean shouldProcess = false;
for( ArbitraryFileListResponseInfo responseInfo : responseInfos) { synchronized (arbitraryDataFileManager.arbitraryDataFileHashResponses) {
if (!arbitraryDataFileManager.arbitraryDataFileHashResponses.isEmpty()) {
if( responseInfo == null ) continue; // Sort by lowest number of node hops first
Comparator<ArbitraryFileListResponseInfo> lowestHopsFirstComparator =
Comparator.comparingInt(ArbitraryFileListResponseInfo::getRequestHops);
arbitraryDataFileManager.arbitraryDataFileHashResponses.sort(lowestHopsFirstComparator);
if (Controller.isStopping()) { Iterator iterator = arbitraryDataFileManager.arbitraryDataFileHashResponses.iterator();
return; while (iterator.hasNext()) {
if (Controller.isStopping()) {
return;
}
ArbitraryFileListResponseInfo responseInfo = (ArbitraryFileListResponseInfo) iterator.next();
if (responseInfo == null) {
iterator.remove();
continue;
}
hash58 = responseInfo.getHash58();
peer = responseInfo.getPeer();
signature58 = responseInfo.getSignature58();
Long timestamp = responseInfo.getTimestamp();
if (now - timestamp >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT || signature58 == null || peer == null) {
// Ignore - to be deleted
iterator.remove();
continue;
}
// Skip if already requesting, but don't remove, as we might want to retry later
if (arbitraryDataFileManager.arbitraryDataFileRequests.containsKey(hash58)) {
// Already requesting - leave this attempt for later
continue;
}
// We want to process this file
shouldProcess = true;
iterator.remove();
break;
}
} }
Peer peer = responseInfo.getPeer();
// if relay timeout, then move on
if (now - responseInfo.getTimestamp() >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT || responseInfo.getSignature58() == null || peer == null) {
continue;
}
// Skip if already requesting, but don't remove, as we might want to retry later
if (arbitraryDataFileManager.arbitraryDataFileRequests.containsKey(responseInfo.getHash58())) {
// Already requesting - leave this attempt for later
arbitraryDataFileManager.addResponse(responseInfo); // don't remove -> adding back, beacause it was removed already above
continue;
}
byte[] hash = Base58.decode(responseInfo.getHash58());
byte[] signature = Base58.decode(responseInfo.getSignature58());
// check for null
if (signature == null || hash == null || peer == null) {
continue;
}
// We want to process this file, store and map data to process later
signatureBySignature58.put(responseInfo.getSignature58(), signature);
responseInfoBySignature58
.computeIfAbsent(responseInfo.getSignature58(), signature58 -> new ArrayList<>())
.add(responseInfo);
} }
// if there are no signatures, then there is nothing to process and nothing query the database if (!shouldProcess) {
if( signatureBySignature58.isEmpty() ) return; // Nothing to do
Thread.sleep(1000L);
return;
}
List<ArbitraryTransactionData> arbitraryTransactionDataList = new ArrayList<>(); byte[] hash = Base58.decode(hash58);
byte[] signature = Base58.decode(signature58);
// Fetch the transaction data // Fetch the transaction data
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
arbitraryTransactionDataList.addAll( ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
ArbitraryTransactionUtils.fetchTransactionDataList(repository, new ArrayList<>(signatureBySignature58.values()))); if (arbitraryTransactionData == null) {
} catch (DataException e) {
LOGGER.warn("Unable to fetch transaction data: {}", e.getMessage());
}
if( !arbitraryTransactionDataList.isEmpty() ) {
long start = System.currentTimeMillis();
for(ArbitraryTransactionData data : arbitraryTransactionDataList ) {
String signature58 = Base58.encode(data.getSignature());
for( ArbitraryFileListResponseInfo responseInfo : responseInfoBySignature58.get(signature58)) {
Runnable fetcher = () -> arbitraryDataFileFetcher(arbitraryDataFileManager, responseInfo, data);
this.executorByPeer
.computeIfAbsent(
responseInfo.getPeer().toString(),
peer -> Executors.newFixedThreadPool(
FETCHER_LIMIT_PER_PEER,
new NamedThreadFactory(FETCHER_THREAD_PREFIX + responseInfo.getPeer().toString(), NORM_PRIORITY)
)
)
.execute(fetcher);
}
}
long timeLapse = System.currentTimeMillis() - start;
}
}
private void arbitraryDataFileFetcher(ArbitraryDataFileManager arbitraryDataFileManager, ArbitraryFileListResponseInfo responseInfo, ArbitraryTransactionData arbitraryTransactionData) {
try {
Long now = NTP.getTime();
if (now - responseInfo.getTimestamp() >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT ) {
Peer peer = responseInfo.getPeer();
String hash58 = responseInfo.getHash58();
String signature58 = responseInfo.getSignature58();
LOGGER.debug("Peer {} version {} didn't fetch data file {} for signature {} due to relay timeout.", peer, peer.getPeersVersionString(), hash58, signature58);
return; return;
} }
arbitraryDataFileManager.fetchArbitraryDataFiles( if (signature == null || hash == null || peer == null || arbitraryTransactionData == null) {
responseInfo.getPeer(), return;
arbitraryTransactionData.getSignature(), }
arbitraryTransactionData,
Arrays.asList(Base58.decode(responseInfo.getHash58())) LOGGER.trace("Fetching file {} from peer {} via request thread...", hash58, peer);
); arbitraryDataFileManager.fetchArbitraryDataFiles(repository, peer, signature, arbitraryTransactionData, Arrays.asList(hash));
} catch (DataException e) { } catch (DataException e) {
LOGGER.warn("Unable to process file hashes: {}", e.getMessage()); LOGGER.debug("Unable to process file hashes: {}", e.getMessage());
} }
} }
} }

View File

@ -10,8 +10,6 @@ import org.qortal.arbitrary.misc.Service;
import org.qortal.controller.Controller; import org.qortal.controller.Controller;
import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.data.transaction.TransactionData; import org.qortal.data.transaction.TransactionData;
import org.qortal.event.DataMonitorEvent;
import org.qortal.event.EventBus;
import org.qortal.network.Network; import org.qortal.network.Network;
import org.qortal.network.Peer; import org.qortal.network.Peer;
import org.qortal.repository.DataException; import org.qortal.repository.DataException;
@ -30,7 +28,6 @@ import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.util.*; import java.util.*;
import java.util.stream.Collectors;
public class ArbitraryDataManager extends Thread { public class ArbitraryDataManager extends Thread {
@ -42,10 +39,10 @@ public class ArbitraryDataManager extends Thread {
private int powDifficulty = 14; // Must not be final, as unit tests need to reduce this value private int powDifficulty = 14; // Must not be final, as unit tests need to reduce this value
/** Request timeout when transferring arbitrary data */ /** Request timeout when transferring arbitrary data */
public static final long ARBITRARY_REQUEST_TIMEOUT = 24 * 1000L; // ms public static final long ARBITRARY_REQUEST_TIMEOUT = 12 * 1000L; // ms
/** Maximum time to hold information about an in-progress relay */ /** Maximum time to hold information about an in-progress relay */
public static final long ARBITRARY_RELAY_TIMEOUT = 120 * 1000L; // ms public static final long ARBITRARY_RELAY_TIMEOUT = 60 * 1000L; // ms
/** Maximum time to hold direct peer connection information */ /** Maximum time to hold direct peer connection information */
public static final long ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT = 2 * 60 * 1000L; // ms public static final long ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT = 2 * 60 * 1000L; // ms
@ -94,7 +91,6 @@ public class ArbitraryDataManager extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data Manager"); Thread.currentThread().setName("Arbitrary Data Manager");
Thread.currentThread().setPriority(NORM_PRIORITY);
// Create data directory in case it doesn't exist yet // Create data directory in case it doesn't exist yet
this.createDataDirectory(); this.createDataDirectory();
@ -198,35 +194,13 @@ public class ArbitraryDataManager extends Thread {
final int limit = 100; final int limit = 100;
int offset = 0; int offset = 0;
List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder;
try (final Repository repository = RepositoryManager.getRepository()) {
if( name == null ) {
allArbitraryTransactionsInDescendingOrder
= repository.getArbitraryRepository()
.getLatestArbitraryTransactions();
}
else {
allArbitraryTransactionsInDescendingOrder
= repository.getArbitraryRepository()
.getLatestArbitraryTransactionsByName(name);
}
} catch( Exception e) {
LOGGER.error(e.getMessage(), e);
allArbitraryTransactionsInDescendingOrder = new ArrayList<>(0);
}
// collect processed transactions in a set to ensure outdated data transactions do not get fetched
Set<ArbitraryTransactionDataHashWrapper> processedTransactions = new HashSet<>();
while (!isStopping) { while (!isStopping) {
Thread.sleep(1000L); Thread.sleep(1000L);
// Any arbitrary transactions we want to fetch data for? // Any arbitrary transactions we want to fetch data for?
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
List<byte[]> signatures = processTransactionsForSignatures(limit, offset, allArbitraryTransactionsInDescendingOrder, processedTransactions); List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, ARBITRARY_TX_TYPE, null, name, null, ConfirmationStatus.BOTH, limit, offset, true);
// LOGGER.trace("Found {} arbitrary transactions at offset: {}, limit: {}", signatures.size(), offset, limit);
if (signatures == null || signatures.isEmpty()) { if (signatures == null || signatures.isEmpty()) {
offset = 0; offset = 0;
break; break;
@ -248,38 +222,14 @@ public class ArbitraryDataManager extends Thread {
ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) arbitraryTransaction.getTransactionData(); ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) arbitraryTransaction.getTransactionData();
// Skip transactions that we don't need to proactively store data for // Skip transactions that we don't need to proactively store data for
ArbitraryDataExamination arbitraryDataExamination = storageManager.shouldPreFetchData(repository, arbitraryTransactionData); if (!storageManager.shouldPreFetchData(repository, arbitraryTransactionData)) {
if (!arbitraryDataExamination.isPass()) {
iterator.remove(); iterator.remove();
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
arbitraryDataExamination.getNotes(),
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
continue; continue;
} }
// Remove transactions that we already have local data for // Remove transactions that we already have local data for
if (hasLocalData(arbitraryTransaction)) { if (hasLocalData(arbitraryTransaction)) {
iterator.remove(); iterator.remove();
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"already have local data, skipping",
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
} }
} }
@ -297,21 +247,8 @@ public class ArbitraryDataManager extends Thread {
// Check to see if we have had a more recent PUT // Check to see if we have had a more recent PUT
ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature); ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
boolean hasMoreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
Optional<ArbitraryTransactionData> moreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData); if (hasMoreRecentPutTransaction) {
if (moreRecentPutTransaction.isPresent()) {
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"not fetching old data",
arbitraryTransactionData.getTimestamp(),
moreRecentPutTransaction.get().getTimestamp()
)
);
// There is a more recent PUT transaction than the one we are currently processing. // There is a more recent PUT transaction than the one we are currently processing.
// When a PUT is issued, it replaces any layers that would have been there before. // When a PUT is issued, it replaces any layers that would have been there before.
// Therefore any data relating to this older transaction is no longer needed and we // Therefore any data relating to this older transaction is no longer needed and we
@ -319,34 +256,10 @@ public class ArbitraryDataManager extends Thread {
continue; continue;
} }
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"fetching data",
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
// Ask our connected peers if they have files for this signature // Ask our connected peers if they have files for this signature
// This process automatically then fetches the files themselves if a peer is found // This process automatically then fetches the files themselves if a peer is found
fetchData(arbitraryTransactionData); fetchData(arbitraryTransactionData);
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"fetched data",
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
} catch (DataException e) { } catch (DataException e) {
LOGGER.error("Repository issue when fetching arbitrary transaction data", e); LOGGER.error("Repository issue when fetching arbitrary transaction data", e);
} }
@ -360,20 +273,6 @@ public class ArbitraryDataManager extends Thread {
final int limit = 100; final int limit = 100;
int offset = 0; int offset = 0;
List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder;
try (final Repository repository = RepositoryManager.getRepository()) {
allArbitraryTransactionsInDescendingOrder
= repository.getArbitraryRepository()
.getLatestArbitraryTransactions();
} catch( Exception e) {
LOGGER.error(e.getMessage(), e);
allArbitraryTransactionsInDescendingOrder = new ArrayList<>(0);
}
// collect processed transactions in a set to ensure outdated data transactions do not get fetched
Set<ArbitraryTransactionDataHashWrapper> processedTransactions = new HashSet<>();
while (!isStopping) { while (!isStopping) {
final int minSeconds = 3; final int minSeconds = 3;
final int maxSeconds = 10; final int maxSeconds = 10;
@ -382,8 +281,8 @@ public class ArbitraryDataManager extends Thread {
// Any arbitrary transactions we want to fetch data for? // Any arbitrary transactions we want to fetch data for?
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
List<byte[]> signatures = processTransactionsForSignatures(limit, offset, allArbitraryTransactionsInDescendingOrder, processedTransactions); List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, ARBITRARY_TX_TYPE, null, null, null, ConfirmationStatus.BOTH, limit, offset, true);
// LOGGER.trace("Found {} arbitrary transactions at offset: {}, limit: {}", signatures.size(), offset, limit);
if (signatures == null || signatures.isEmpty()) { if (signatures == null || signatures.isEmpty()) {
offset = 0; offset = 0;
break; break;
@ -428,74 +327,26 @@ public class ArbitraryDataManager extends Thread {
continue; continue;
} }
// No longer need to see if we have had a more recent PUT since we compared the transactions to process // Check to see if we have had a more recent PUT
// to the transactions previously processed, so we can fetch the transactiondata, notify the event bus,
// fetch the metadata and notify the event bus again
ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature); ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
boolean hasMoreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
if (hasMoreRecentPutTransaction) {
// There is a more recent PUT transaction than the one we are currently processing.
// When a PUT is issued, it replaces any layers that would have been there before.
// Therefore any data relating to this older transaction is no longer needed and we
// shouldn't fetch it from the network.
continue;
}
// Ask our connected peers if they have metadata for this signature // Ask our connected peers if they have metadata for this signature
fetchMetadata(arbitraryTransactionData); fetchMetadata(arbitraryTransactionData);
EventBus.INSTANCE.notify(
new DataMonitorEvent(
System.currentTimeMillis(),
arbitraryTransactionData.getIdentifier(),
arbitraryTransactionData.getName(),
arbitraryTransactionData.getService().name(),
"fetched metadata",
arbitraryTransactionData.getTimestamp(),
arbitraryTransactionData.getTimestamp()
)
);
} catch (DataException e) { } catch (DataException e) {
LOGGER.error("Repository issue when fetching arbitrary transaction data", e); LOGGER.error("Repository issue when fetching arbitrary transaction data", e);
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
} }
} }
} }
private static List<byte[]> processTransactionsForSignatures(
int limit,
int offset,
List<ArbitraryTransactionData> transactionsInDescendingOrder,
Set<ArbitraryTransactionDataHashWrapper> processedTransactions) {
// these transactions are in descending order, latest transactions come first
List<ArbitraryTransactionData> transactions
= transactionsInDescendingOrder.stream()
.skip(offset)
.limit(limit)
.collect(Collectors.toList());
// wrap the transactions, so they can be used for hashing and comparing
// Class ArbitraryTransactionDataHashWrapper supports hashCode() and equals(...) for this purpose
List<ArbitraryTransactionDataHashWrapper> wrappedTransactions
= transactions.stream()
.map(transaction -> new ArbitraryTransactionDataHashWrapper(transaction))
.collect(Collectors.toList());
// create a set of wrappers and populate it first to last, so that all outdated transactions get rejected
Set<ArbitraryTransactionDataHashWrapper> transactionsToProcess = new HashSet<>(wrappedTransactions.size());
for(ArbitraryTransactionDataHashWrapper wrappedTransaction : wrappedTransactions) {
transactionsToProcess.add(wrappedTransaction);
}
// remove the matches for previously processed transactions,
// because these transactions have had updates that have already been processed
transactionsToProcess.removeAll(processedTransactions);
// add to processed transactions to compare and remove matches from future processing iterations
processedTransactions.addAll(transactionsToProcess);
List<byte[]> signatures
= transactionsToProcess.stream()
.map(transactionToProcess -> transactionToProcess.getData()
.getSignature())
.collect(Collectors.toList());
return signatures;
}
private ArbitraryTransaction fetchTransaction(final Repository repository, byte[] signature) { private ArbitraryTransaction fetchTransaction(final Repository repository, byte[] signature) {
try { try {
TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature); TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);

View File

@ -36,7 +36,6 @@ public class ArbitraryDataRenderManager extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data Render Manager"); Thread.currentThread().setName("Arbitrary Data Render Manager");
Thread.currentThread().setPriority(NORM_PRIORITY);
try { try {
while (!isStopping) { while (!isStopping) {

View File

@ -47,15 +47,15 @@ public class ArbitraryDataStorageManager extends Thread {
private static final long DIRECTORY_SIZE_CHECK_INTERVAL = 10 * 60 * 1000L; // 10 minutes private static final long DIRECTORY_SIZE_CHECK_INTERVAL = 10 * 60 * 1000L; // 10 minutes
/** Treat storage as full at 80% usage, to reduce risk of going over the limit. /** Treat storage as full at 90% usage, to reduce risk of going over the limit.
* This is necessary because we don't calculate total storage values before every write. * This is necessary because we don't calculate total storage values before every write.
* It also helps avoid a fetch/delete loop, as we will stop fetching before the hard limit. * It also helps avoid a fetch/delete loop, as we will stop fetching before the hard limit.
* This must be lower than DELETION_THRESHOLD. */ * This must be lower than DELETION_THRESHOLD. */
private static final double STORAGE_FULL_THRESHOLD = 0.8f; // 80% private static final double STORAGE_FULL_THRESHOLD = 0.90f; // 90%
/** Start deleting files once we reach 90% usage. /** Start deleting files once we reach 98% usage.
* This must be higher than STORAGE_FULL_THRESHOLD in order to avoid a fetch/delete loop. */ * This must be higher than STORAGE_FULL_THRESHOLD in order to avoid a fetch/delete loop. */
public static final double DELETION_THRESHOLD = 0.9f; // 90% public static final double DELETION_THRESHOLD = 0.98f; // 98%
private static final long PER_NAME_STORAGE_MULTIPLIER = 4L; private static final long PER_NAME_STORAGE_MULTIPLIER = 4L;
@ -72,8 +72,6 @@ public class ArbitraryDataStorageManager extends Thread {
@Override @Override
public void run() { public void run() {
Thread.currentThread().setName("Arbitrary Data Storage Manager"); Thread.currentThread().setName("Arbitrary Data Storage Manager");
Thread.currentThread().setPriority(NORM_PRIORITY);
try { try {
while (!isStopping) { while (!isStopping) {
Thread.sleep(1000); Thread.sleep(1000);
@ -155,24 +153,31 @@ public class ArbitraryDataStorageManager extends Thread {
* @param arbitraryTransactionData - the transaction * @param arbitraryTransactionData - the transaction
* @return boolean - whether to prefetch or not * @return boolean - whether to prefetch or not
*/ */
public ArbitraryDataExamination shouldPreFetchData(Repository repository, ArbitraryTransactionData arbitraryTransactionData) { public boolean shouldPreFetchData(Repository repository, ArbitraryTransactionData arbitraryTransactionData) {
String name = arbitraryTransactionData.getName(); String name = arbitraryTransactionData.getName();
// Only fetch data associated with hashes, as we already have RAW_DATA // Only fetch data associated with hashes, as we already have RAW_DATA
if (arbitraryTransactionData.getDataType() != ArbitraryTransactionData.DataType.DATA_HASH) { if (arbitraryTransactionData.getDataType() != ArbitraryTransactionData.DataType.DATA_HASH) {
return new ArbitraryDataExamination(false, "Only fetch data associated with hashes"); return false;
} }
// Don't fetch anything more if we're (nearly) out of space // Don't fetch anything more if we're (nearly) out of space
// Make sure to keep STORAGE_FULL_THRESHOLD considerably less than 1, to // Make sure to keep STORAGE_FULL_THRESHOLD considerably less than 1, to
// avoid a fetch/delete loop // avoid a fetch/delete loop
if (!this.isStorageSpaceAvailable(STORAGE_FULL_THRESHOLD)) { if (!this.isStorageSpaceAvailable(STORAGE_FULL_THRESHOLD)) {
return new ArbitraryDataExamination(false,"Don't fetch anything more if we're (nearly) out of space"); return false;
}
// Don't fetch anything if we're (nearly) out of space for this name
// Again, make sure to keep STORAGE_FULL_THRESHOLD considerably less than 1, to
// avoid a fetch/delete loop
if (!this.isStorageSpaceAvailableForName(repository, arbitraryTransactionData.getName(), STORAGE_FULL_THRESHOLD)) {
return false;
} }
// Don't store data unless it's an allowed type (public/private) // Don't store data unless it's an allowed type (public/private)
if (!this.isDataTypeAllowed(arbitraryTransactionData)) { if (!this.isDataTypeAllowed(arbitraryTransactionData)) {
return new ArbitraryDataExamination(false, "Don't store data unless it's an allowed type (public/private)"); return false;
} }
// Handle transactions without names differently // Handle transactions without names differently
@ -182,21 +187,21 @@ public class ArbitraryDataStorageManager extends Thread {
// Never fetch data from blocked names, even if they are followed // Never fetch data from blocked names, even if they are followed
if (ListUtils.isNameBlocked(name)) { if (ListUtils.isNameBlocked(name)) {
return new ArbitraryDataExamination(false, "blocked name"); return false;
} }
switch (Settings.getInstance().getStoragePolicy()) { switch (Settings.getInstance().getStoragePolicy()) {
case FOLLOWED: case FOLLOWED:
case FOLLOWED_OR_VIEWED: case FOLLOWED_OR_VIEWED:
return new ArbitraryDataExamination(ListUtils.isFollowingName(name), Settings.getInstance().getStoragePolicy().name()); return ListUtils.isFollowingName(name);
case ALL: case ALL:
return new ArbitraryDataExamination(true, Settings.getInstance().getStoragePolicy().name()); return true;
case NONE: case NONE:
case VIEWED: case VIEWED:
default: default:
return new ArbitraryDataExamination(false, Settings.getInstance().getStoragePolicy().name()); return false;
} }
} }
@ -207,17 +212,17 @@ public class ArbitraryDataStorageManager extends Thread {
* *
* @return boolean - whether the storage policy allows for unnamed data * @return boolean - whether the storage policy allows for unnamed data
*/ */
private ArbitraryDataExamination shouldPreFetchDataWithoutName() { private boolean shouldPreFetchDataWithoutName() {
switch (Settings.getInstance().getStoragePolicy()) { switch (Settings.getInstance().getStoragePolicy()) {
case ALL: case ALL:
return new ArbitraryDataExamination(true, "Fetching all data"); return true;
case NONE: case NONE:
case VIEWED: case VIEWED:
case FOLLOWED: case FOLLOWED:
case FOLLOWED_OR_VIEWED: case FOLLOWED_OR_VIEWED:
default: default:
return new ArbitraryDataExamination(false, Settings.getInstance().getStoragePolicy().name()); return false;
} }
} }
@ -477,6 +482,51 @@ public class ArbitraryDataStorageManager extends Thread {
return true; return true;
} }
public boolean isStorageSpaceAvailableForName(Repository repository, String name, double threshold) {
if (!this.isStorageSpaceAvailable(threshold)) {
// No storage space available at all, so no need to check this name
return false;
}
if (Settings.getInstance().getStoragePolicy() == StoragePolicy.ALL) {
// Using storage policy ALL, so don't limit anything per name
return true;
}
if (name == null) {
// This transaction doesn't have a name, so fall back to total space limitations
return true;
}
int followedNamesCount = ListUtils.followedNamesCount();
if (followedNamesCount == 0) {
// Not following any names, so we have space
return true;
}
long totalSizeForName = 0;
long maxStoragePerName = this.storageCapacityPerName(threshold);
// Fetch all hosted transactions
List<ArbitraryTransactionData> hostedTransactions = this.listAllHostedTransactions(repository, null, null);
for (ArbitraryTransactionData transactionData : hostedTransactions) {
String transactionName = transactionData.getName();
if (!Objects.equals(name, transactionName)) {
// Transaction relates to a different name
continue;
}
totalSizeForName += transactionData.getSize();
}
// Have we reached the limit for this name?
if (totalSizeForName > maxStoragePerName) {
return false;
}
return true;
}
public long storageCapacityPerName(double threshold) { public long storageCapacityPerName(double threshold) {
int followedNamesCount = ListUtils.followedNamesCount(); int followedNamesCount = ListUtils.followedNamesCount();
if (followedNamesCount == 0) { if (followedNamesCount == 0) {

View File

@ -24,11 +24,6 @@ import org.qortal.utils.Triple;
import java.io.IOException; import java.io.IOException;
import java.util.*; import java.util.*;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.stream.Collectors;
import static org.qortal.controller.arbitrary.ArbitraryDataFileListManager.*; import static org.qortal.controller.arbitrary.ArbitraryDataFileListManager.*;
@ -66,7 +61,6 @@ public class ArbitraryMetadataManager {
private ArbitraryMetadataManager() { private ArbitraryMetadataManager() {
scheduler.scheduleAtFixedRate(this::processNetworkGetArbitraryMetadataMessage, 60, 1, TimeUnit.SECONDS);
} }
public static ArbitraryMetadataManager getInstance() { public static ArbitraryMetadataManager getInstance() {
@ -360,8 +354,9 @@ public class ArbitraryMetadataManager {
// Forward to requesting peer // Forward to requesting peer
LOGGER.debug("Forwarding metadata to requesting peer: {}", requestingPeer); LOGGER.debug("Forwarding metadata to requesting peer: {}", requestingPeer);
requestingPeer.sendMessage(forwardArbitraryMetadataMessage); if (!requestingPeer.sendMessage(forwardArbitraryMetadataMessage)) {
requestingPeer.disconnect("failed to forward arbitrary metadata");
}
} }
} }
} }
@ -376,159 +371,107 @@ public class ArbitraryMetadataManager {
} }
} }
// List to collect messages
private final List<PeerMessage> messageList = new ArrayList<>();
// Lock to synchronize access to the list
private final Object lock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
public void onNetworkGetArbitraryMetadataMessage(Peer peer, Message message) { public void onNetworkGetArbitraryMetadataMessage(Peer peer, Message message) {
// Don't respond if QDN is disabled // Don't respond if QDN is disabled
if (!Settings.getInstance().isQdnEnabled()) { if (!Settings.getInstance().isQdnEnabled()) {
return; return;
} }
synchronized (lock) { Controller.getInstance().stats.getArbitraryMetadataMessageStats.requests.incrementAndGet();
messageList.add(new PeerMessage(peer, message));
GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) message;
byte[] signature = getArbitraryMetadataMessage.getSignature();
String signature58 = Base58.encode(signature);
Long now = NTP.getTime();
Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peer, now);
// If we've seen this request recently, then ignore
if (arbitraryMetadataRequests.putIfAbsent(message.getId(), newEntry) != null) {
LOGGER.debug("Ignoring metadata request from peer {} for signature {}", peer, signature58);
return;
} }
}
private void processNetworkGetArbitraryMetadataMessage() { LOGGER.debug("Received metadata request from peer {} for signature {}", peer, signature58);
try { ArbitraryTransactionData transactionData = null;
List<PeerMessage> messagesToProcess; ArbitraryDataFile metadataFile = null;
synchronized (lock) {
messagesToProcess = new ArrayList<>(messageList);
messageList.clear();
}
Map<String, byte[]> signatureBySignature58 = new HashMap<>((messagesToProcess.size())); try (final Repository repository = RepositoryManager.getRepository()) {
Map<String, Long> nowBySignature58 = new HashMap<>(messagesToProcess.size());
Map<String,PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
for( PeerMessage peerMessage : messagesToProcess) { // Firstly we need to lookup this file on chain to get its metadata hash
Controller.getInstance().stats.getArbitraryMetadataMessageStats.requests.incrementAndGet(); transactionData = (ArbitraryTransactionData)repository.getTransactionRepository().fromSignature(signature);
if (transactionData instanceof ArbitraryTransactionData) {
GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) peerMessage.message; // Check if we're even allowed to serve metadata for this transaction
byte[] signature = getArbitraryMetadataMessage.getSignature(); if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
String signature58 = Base58.encode(signature);
Long now = NTP.getTime();
Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peerMessage.peer, now);
// If we've seen this request recently, then ignore byte[] metadataHash = transactionData.getMetadataHash();
if (arbitraryMetadataRequests.putIfAbsent(peerMessage.message.getId(), newEntry) != null) { if (metadataHash != null) {
LOGGER.debug("Ignoring metadata request from peer {} for signature {}", peerMessage.peer, signature58);
continue; // Load metadata file
metadataFile = ArbitraryDataFile.fromHash(metadataHash, signature);
}
} }
LOGGER.debug("Received metadata request from peer {} for signature {}", peerMessage.peer, signature58);
signatureBySignature58.put(signature58, signature);
nowBySignature58.put(signature58, now);
peerMessageBySignature58.put(signature58, peerMessage);
} }
if( signatureBySignature58.isEmpty() ) return; } catch (DataException e) {
LOGGER.error(String.format("Repository issue while fetching arbitrary metadata for peer %s", peer), e);
}
List<TransactionData> transactionDataList; // We should only respond if we have the metadata file
try (final Repository repository = RepositoryManager.getRepository()) { if (metadataFile != null && metadataFile.exists()) {
// Firstly we need to lookup this file on chain to get its metadata hash // We have the metadata file, so update requests map to reflect that we've sent it
transactionDataList = repository.getTransactionRepository().fromSignatures(new ArrayList(signatureBySignature58.values())); newEntry = new Triple<>(null, null, now);
} catch (DataException e) { arbitraryMetadataRequests.put(message.getId(), newEntry);
LOGGER.error(String.format("Repository issue while fetching arbitrary transactions"), e);
ArbitraryMetadataMessage arbitraryMetadataMessage = new ArbitraryMetadataMessage(signature, metadataFile);
arbitraryMetadataMessage.setId(message.getId());
if (!peer.sendMessage(arbitraryMetadataMessage)) {
LOGGER.debug("Couldn't send metadata");
peer.disconnect("failed to send metadata");
return; return;
} }
LOGGER.debug("Sent metadata");
Map<String, ArbitraryTransactionData> dataBySignature58 // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
= transactionDataList.stream() LOGGER.debug("No need for any forwarding because metadata request is fully served");
.filter(data -> data instanceof ArbitraryTransactionData) return;
.map(ArbitraryTransactionData.class::cast)
.collect(Collectors.toMap(data -> Base58.encode(data.getSignature()), Function.identity()));
for(Map.Entry<String, ArbitraryTransactionData> entry : dataBySignature58.entrySet()) { }
String signature58 = entry.getKey();
ArbitraryTransactionData transactionData = entry.getValue();
try { // We may need to forward this request on
boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
// In relay mode - so ask our other peers if they have it
// Check if we're even allowed to serve metadata for this transaction long requestTime = getArbitraryMetadataMessage.getRequestTime();
if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) { int requestHops = getArbitraryMetadataMessage.getRequestHops() + 1;
long totalRequestTime = now - requestTime;
byte[] metadataHash = transactionData.getMetadataHash(); if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
if (metadataHash != null) { // Relay request hasn't timed out yet, so can potentially be rebroadcast
if (requestHops < RELAY_REQUEST_MAX_HOPS) {
// Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
// Load metadata file Message relayGetArbitraryMetadataMessage = new GetArbitraryMetadataMessage(signature, requestTime, requestHops);
ArbitraryDataFile metadataFile = ArbitraryDataFile.fromHash(metadataHash, transactionData.getSignature()); relayGetArbitraryMetadataMessage.setId(message.getId());
// We should only respond if we have the metadata file
if (metadataFile != null && metadataFile.exists()) {
PeerMessage peerMessage = peerMessageBySignature58.get(signature58); LOGGER.debug("Rebroadcasting metadata request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
Message message = peerMessage.message; Network.getInstance().broadcast(
Peer peer = peerMessage.peer; broadcastPeer ->
!broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryMetadataMessage);
// We have the metadata file, so update requests map to reflect that we've sent it
Triple newEntry = new Triple<>(null, null, nowBySignature58.get(signature58));
arbitraryMetadataRequests.put(message.getId(), newEntry);
ArbitraryMetadataMessage arbitraryMetadataMessage = new ArbitraryMetadataMessage(entry.getValue().getSignature(), metadataFile);
arbitraryMetadataMessage.setId(message.getId());
if (!peer.sendMessage(arbitraryMetadataMessage)) {
LOGGER.debug("Couldn't send metadata");
continue;
}
LOGGER.debug("Sent metadata");
// Nothing left to do, so return to prevent any unnecessary forwarding from occurring
LOGGER.debug("No need for any forwarding because metadata request is fully served");
}
}
}
} catch (DataException e) {
LOGGER.error(String.format("Repository issue while fetching arbitrary metadata"), e);
} }
else {
// We may need to forward this request on // This relay request has reached the maximum number of allowed hops
boolean isBlocked = (transactionDataList == null || ListUtils.isNameBlocked(transactionData.getName()));
if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
// In relay mode - so ask our other peers if they have it
PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) peerMessage.message;
long requestTime = getArbitraryMetadataMessage.getRequestTime();
int requestHops = getArbitraryMetadataMessage.getRequestHops() + 1;
long totalRequestTime = nowBySignature58.get(signature58) - requestTime;
if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
// Relay request hasn't timed out yet, so can potentially be rebroadcast
if (requestHops < RELAY_REQUEST_MAX_HOPS) {
// Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
byte[] signature = signatureBySignature58.get(signature58);
Message relayGetArbitraryMetadataMessage = new GetArbitraryMetadataMessage(signature, requestTime, requestHops);
relayGetArbitraryMetadataMessage.setId(getArbitraryMetadataMessage.getId());
Peer peer = peerMessage.peer;
LOGGER.debug("Rebroadcasting metadata request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
Network.getInstance().broadcast(
broadcastPeer ->
!broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryMetadataMessage);
} else {
// This relay request has reached the maximum number of allowed hops
}
} else {
// This relay request has timed out
}
} }
} }
} catch (Exception e) { else {
LOGGER.error(e.getMessage(), e); // This relay request has timed out
}
} }
} }
} }

View File

@ -1,48 +0,0 @@
package org.qortal.controller.arbitrary;
import org.qortal.arbitrary.misc.Service;
import org.qortal.data.transaction.ArbitraryTransactionData;
import java.util.Objects;
public class ArbitraryTransactionDataHashWrapper {
private ArbitraryTransactionData data;
private int service;
private String name;
private String identifier;
public ArbitraryTransactionDataHashWrapper(ArbitraryTransactionData data) {
this.data = data;
this.service = data.getService().value;
this.name = data.getName();
this.identifier = data.getIdentifier();
}
public ArbitraryTransactionDataHashWrapper(int service, String name, String identifier) {
this.service = service;
this.name = name;
this.identifier = identifier;
}
public ArbitraryTransactionData getData() {
return data;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ArbitraryTransactionDataHashWrapper that = (ArbitraryTransactionDataHashWrapper) o;
return service == that.service && name.equals(that.name) && Objects.equals(identifier, that.identifier);
}
@Override
public int hashCode() {
return Objects.hash(service, name, identifier);
}
}

View File

@ -1,130 +0,0 @@
package org.qortal.controller.arbitrary;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.utils.ListUtils;
import org.qortal.utils.NamedThreadFactory;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.OptionalInt;
import java.util.Set;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
public class Follower {
private static final Logger LOGGER = LogManager.getLogger(Follower.class);
private ScheduledExecutorService service
= Executors.newScheduledThreadPool(2, new NamedThreadFactory("Follower", Thread.NORM_PRIORITY));
private Follower() {
}
private static Follower instance;
public static Follower getInstance() {
if( instance == null ) {
instance = new Follower();
}
return instance;
}
public void start() {
// fetch arbitrary transactions from followed names from the last 100 blocks every 2 minutes
service.scheduleWithFixedDelay(() -> fetch(OptionalInt.of(100)), 10, 2, TimeUnit.MINUTES);
// fetch arbitrary transaction from followed names from any block every 24 hours
service.scheduleWithFixedDelay(() -> fetch(OptionalInt.empty()), 4, 24, TimeUnit.HOURS);
}
private void fetch(OptionalInt limit) {
try {
// for each followed name, get arbitraty transactions, then examine those transactions before fetching
for (String name : ListUtils.followedNames()) {
List<ArbitraryTransactionData> transactionsInReverseOrder;
// open database to get the transactions in reverse order for the followed name
try (final Repository repository = RepositoryManager.getRepository()) {
List<ArbitraryTransactionData> latestArbitraryTransactionsByName
= repository.getArbitraryRepository().getLatestArbitraryTransactionsByName(name);
if (limit.isPresent()) {
final int blockHeightThreshold = repository.getBlockRepository().getBlockchainHeight() - limit.getAsInt();
transactionsInReverseOrder
= latestArbitraryTransactionsByName.stream().filter(tx -> tx.getBlockHeight() > blockHeightThreshold)
.collect(Collectors.toList());
} else {
transactionsInReverseOrder = latestArbitraryTransactionsByName;
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
transactionsInReverseOrder = new ArrayList<>(0);
}
// collect process transaction hashes, so we don't fetch outdated transactions
Set<ArbitraryTransactionDataHashWrapper> processedTransactions = new HashSet<>();
ArbitraryDataStorageManager storageManager = ArbitraryDataStorageManager.getInstance();
// for each arbitrary transaction for the followed name process, evaluate, fetch
for (ArbitraryTransactionData arbitraryTransaction : transactionsInReverseOrder) {
boolean examined = false;
try (final Repository repository = RepositoryManager.getRepository()) {
// if not processed
if (!processedTransactions.contains(new ArbitraryTransactionDataHashWrapper(arbitraryTransaction))) {
boolean isLocal = repository.getArbitraryRepository().isDataLocal(arbitraryTransaction.getSignature());
// if not local, then continue to evaluate
if (!isLocal) {
// evaluate fetching status for this transaction on this node
ArbitraryDataExamination examination = storageManager.shouldPreFetchData(repository, arbitraryTransaction);
// if the evaluation passed, then fetch
examined = examination.isPass();
}
// if locally stored, then nothing needs to be done
// add to processed transactions
processedTransactions.add(new ArbitraryTransactionDataHashWrapper(arbitraryTransaction));
}
}
// if passed examination for fetching, then fetch
if (examined) {
LOGGER.info("for {} on {}, fetching {}", name, arbitraryTransaction.getService(), arbitraryTransaction.getIdentifier());
boolean fetched = ArbitraryDataFileListManager.getInstance().fetchArbitraryDataFileList(arbitraryTransaction);
LOGGER.info("fetched = " + fetched);
}
// pause a second before moving on to another transaction
Thread.sleep(1000);
}
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
}

View File

@ -1,22 +0,0 @@
package org.qortal.controller.arbitrary;
import org.qortal.network.Peer;
import org.qortal.network.message.Message;
public class PeerMessage {
Peer peer;
Message message;
public PeerMessage(Peer peer, Message message) {
this.peer = peer;
this.message = message;
}
public Peer getPeer() {
return peer;
}
public Message getMessage() {
return message;
}
}

View File

@ -1,33 +0,0 @@
package org.qortal.controller.arbitrary;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import java.util.TimerTask;
public class RebuildArbitraryResourceCacheTask extends TimerTask {
private static final Logger LOGGER = LogManager.getLogger(RebuildArbitraryResourceCacheTask.class);
public static final long MILLIS_IN_HOUR = 60 * 60 * 1000;
public static final long MILLIS_IN_MINUTE = 60 * 1000;
private static final String REBUILD_ARBITRARY_RESOURCE_CACHE_TASK = "Rebuild Arbitrary Resource Cache Task";
@Override
public void run() {
Thread.currentThread().setName(REBUILD_ARBITRARY_RESOURCE_CACHE_TASK);
try (final Repository repository = RepositoryManager.getRepository()) {
ArbitraryDataCacheManager.getInstance().buildArbitraryResourcesCache(repository, true);
}
catch( DataException e ) {
LOGGER.error(e.getMessage(), e);
}
}
}

View File

@ -1,139 +0,0 @@
package org.qortal.controller.hsqldb;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.util.PropertySource;
import org.qortal.data.account.AccountBalanceData;
import org.qortal.data.account.BlockHeightRange;
import org.qortal.data.account.BlockHeightRangeAddressAmounts;
import org.qortal.repository.hsqldb.HSQLDBCacheUtils;
import org.qortal.settings.Settings;
import org.qortal.utils.BalanceRecorderUtils;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.stream.Collectors;
public class HSQLDBBalanceRecorder extends Thread{
private static final Logger LOGGER = LogManager.getLogger(HSQLDBBalanceRecorder.class);
private static HSQLDBBalanceRecorder SINGLETON = null;
private ConcurrentHashMap<Integer, List<AccountBalanceData>> balancesByHeight = new ConcurrentHashMap<>();
private ConcurrentHashMap<String, List<AccountBalanceData>> balancesByAddress = new ConcurrentHashMap<>();
private CopyOnWriteArrayList<BlockHeightRangeAddressAmounts> balanceDynamics = new CopyOnWriteArrayList<>();
private int priorityRequested;
private int frequency;
private int capacity;
private HSQLDBBalanceRecorder( int priorityRequested, int frequency, int capacity) {
super("Balance Recorder");
this.priorityRequested = priorityRequested;
this.frequency = frequency;
this.capacity = capacity;
}
public static Optional<HSQLDBBalanceRecorder> getInstance() {
if( SINGLETON == null ) {
SINGLETON
= new HSQLDBBalanceRecorder(
Settings.getInstance().getBalanceRecorderPriority(),
Settings.getInstance().getBalanceRecorderFrequency(),
Settings.getInstance().getBalanceRecorderCapacity()
);
}
else if( SINGLETON == null ) {
return Optional.empty();
}
return Optional.of(SINGLETON);
}
@Override
public void run() {
Thread.currentThread().setName("Balance Recorder");
HSQLDBCacheUtils.startRecordingBalances(this.balancesByHeight, this.balanceDynamics, this.priorityRequested, this.frequency, this.capacity);
}
public List<BlockHeightRangeAddressAmounts> getLatestDynamics(int limit, long offset) {
List<BlockHeightRangeAddressAmounts> latest = this.balanceDynamics.stream()
.sorted(BalanceRecorderUtils.BLOCK_HEIGHT_RANGE_ADDRESS_AMOUNTS_COMPARATOR.reversed())
.skip(offset)
.limit(limit)
.collect(Collectors.toList());
return latest;
}
public List<BlockHeightRange> getRanges(Integer offset, Integer limit, Boolean reverse) {
if( reverse ) {
return this.balanceDynamics.stream()
.map(BlockHeightRangeAddressAmounts::getRange)
.sorted(BalanceRecorderUtils.BLOCK_HEIGHT_RANGE_COMPARATOR.reversed())
.skip(offset)
.limit(limit)
.collect(Collectors.toList());
}
else {
return this.balanceDynamics.stream()
.map(BlockHeightRangeAddressAmounts::getRange)
.sorted(BalanceRecorderUtils.BLOCK_HEIGHT_RANGE_COMPARATOR)
.skip(offset)
.limit(limit)
.collect(Collectors.toList());
}
}
public Optional<BlockHeightRangeAddressAmounts> getAddressAmounts(BlockHeightRange range) {
return this.balanceDynamics.stream()
.filter( dynamic -> dynamic.getRange().equals(range))
.findAny();
}
public Optional<BlockHeightRange> getRange( int height ) {
return this.balanceDynamics.stream()
.map(BlockHeightRangeAddressAmounts::getRange)
.filter( range -> range.getBegin() < height && range.getEnd() >= height )
.findAny();
}
private Optional<Integer> getLastHeight() {
return this.balancesByHeight.keySet().stream().sorted(Comparator.reverseOrder()).findFirst();
}
public List<Integer> getBlocksRecorded() {
return this.balancesByHeight.keySet().stream().collect(Collectors.toList());
}
public List<AccountBalanceData> getAccountBalanceRecordings(String address) {
return this.balancesByAddress.get(address);
}
@Override
public String toString() {
return "HSQLDBBalanceRecorder{" +
"priorityRequested=" + priorityRequested +
", frequency=" + frequency +
", capacity=" + capacity +
'}';
}
}

View File

@ -1,22 +0,0 @@
package org.qortal.controller.hsqldb;
import org.qortal.data.arbitrary.ArbitraryResourceCache;
import org.qortal.repository.RepositoryManager;
import org.qortal.repository.hsqldb.HSQLDBCacheUtils;
import org.qortal.repository.hsqldb.HSQLDBRepository;
import org.qortal.settings.Settings;
public class HSQLDBDataCacheManager extends Thread{
public HSQLDBDataCacheManager() {}
@Override
public void run() {
Thread.currentThread().setName("HSQLDB Data Cache Manager");
HSQLDBCacheUtils.startCaching(
Settings.getInstance().getDbCacheThreadPriority(),
Settings.getInstance().getDbCacheFrequency()
);
}
}

View File

@ -11,8 +11,6 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import static java.lang.Thread.MIN_PRIORITY;
public class AtStatesPruner implements Runnable { public class AtStatesPruner implements Runnable {
private static final Logger LOGGER = LogManager.getLogger(AtStatesPruner.class); private static final Logger LOGGER = LogManager.getLogger(AtStatesPruner.class);
@ -39,97 +37,82 @@ public class AtStatesPruner implements Runnable {
} }
} }
int pruneStartHeight;
int maxLatestAtStatesHeight;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
pruneStartHeight = repository.getATRepository().getAtPruneHeight(); int pruneStartHeight = repository.getATRepository().getAtPruneHeight();
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository); int maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.discardChanges(); repository.discardChanges();
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight); repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges(); repository.saveChanges();
} catch (Exception e) {
LOGGER.error("AT States Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
return;
}
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
try (final Repository repository = RepositoryManager.getRepository()) { repository.discardChanges();
try { Thread.sleep(Settings.getInstance().getAtStatesPruneInterval());
repository.discardChanges();
Thread.sleep(Settings.getInstance().getAtStatesPruneInterval()); BlockData chainTip = Controller.getInstance().getChainTip();
if (chainTip == null || NTP.getTime() == null)
continue;
BlockData chainTip = Controller.getInstance().getChainTip(); // Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (chainTip == null || NTP.getTime() == null) if (Synchronizer.getInstance().isSynchronizing())
continue; continue;
// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages // Prune AT states for all blocks up until our latest minus pruneBlockLimit
if (Synchronizer.getInstance().isSynchronizing()) final int ourLatestHeight = chainTip.getHeight();
continue; int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
// Prune AT states for all blocks up until our latest minus pruneBlockLimit // In archive mode we are only allowed to trim blocks that have already been archived
final int ourLatestHeight = chainTip.getHeight(); if (archiveMode) {
int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit(); upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
// In archive mode we are only allowed to trim blocks that have already been archived // TODO: validate that the actual archived data exists before pruning it?
if (archiveMode) { }
upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
int upperBatchHeight = pruneStartHeight + Settings.getInstance().getAtStatesPruneBatchSize();
// TODO: validate that the actual archived data exists before pruning it? int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
}
if (pruneStartHeight >= upperPruneHeight)
int upperBatchHeight = pruneStartHeight + Settings.getInstance().getAtStatesPruneBatchSize(); continue;
int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
LOGGER.debug(String.format("Pruning AT states between blocks %d and %d...", pruneStartHeight, upperPruneHeight));
if (pruneStartHeight >= upperPruneHeight)
continue; int numAtStatesPruned = repository.getATRepository().pruneAtStates(pruneStartHeight, upperPruneHeight);
repository.saveChanges();
LOGGER.info(String.format("Pruning AT states between blocks %d and %d...", pruneStartHeight, upperPruneHeight)); int numAtStateDataRowsTrimmed = repository.getATRepository().trimAtStates(
pruneStartHeight, upperPruneHeight, Settings.getInstance().getAtStatesTrimLimit());
int numAtStatesPruned = repository.getATRepository().pruneAtStates(pruneStartHeight, upperPruneHeight); repository.saveChanges();
repository.saveChanges();
int numAtStateDataRowsTrimmed = repository.getATRepository().trimAtStates( if (numAtStatesPruned > 0 || numAtStateDataRowsTrimmed > 0) {
pruneStartHeight, upperPruneHeight, Settings.getInstance().getAtStatesTrimLimit()); final int finalPruneStartHeight = pruneStartHeight;
repository.saveChanges(); LOGGER.debug(() -> String.format("Pruned %d AT state%s between blocks %d and %d",
numAtStatesPruned, (numAtStatesPruned != 1 ? "s" : ""),
if (numAtStatesPruned > 0 || numAtStateDataRowsTrimmed > 0) { finalPruneStartHeight, upperPruneHeight));
final int finalPruneStartHeight = pruneStartHeight; } else {
LOGGER.info(() -> String.format("Pruned %d AT state%s between blocks %d and %d", // Can we move onto next batch?
numAtStatesPruned, (numAtStatesPruned != 1 ? "s" : ""), if (upperPrunableHeight > upperBatchHeight) {
finalPruneStartHeight, upperPruneHeight)); pruneStartHeight = upperBatchHeight;
} else { repository.getATRepository().setAtPruneHeight(pruneStartHeight);
// Can we move onto next batch? maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
if (upperPrunableHeight > upperBatchHeight) { repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
pruneStartHeight = upperBatchHeight; repository.saveChanges();
repository.getATRepository().setAtPruneHeight(pruneStartHeight);
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository); final int finalPruneStartHeight = pruneStartHeight;
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight); LOGGER.debug(() -> String.format("Bumping AT state base prune height to %d", finalPruneStartHeight));
repository.saveChanges(); }
else {
final int finalPruneStartHeight = pruneStartHeight; // We've pruned up to the upper prunable height
LOGGER.info(() -> String.format("Bumping AT state base prune height to %d", finalPruneStartHeight)); // Back off for a while to save CPU for syncing
} else { repository.discardChanges();
// We've pruned up to the upper prunable height Thread.sleep(5*60*1000L);
// Back off for a while to save CPU for syncing }
repository.discardChanges();
Thread.sleep(5 * 60 * 1000L);
}
}
} catch (InterruptedException e) {
if (Controller.isStopping()) {
LOGGER.info("AT States Pruning Shutting Down");
} else {
LOGGER.warn("AT States Pruning interrupted. Trying again. Report this error immediately to the developers.", e);
}
} catch (Exception e) {
LOGGER.warn("AT States Pruning stopped working. Trying again. Report this error immediately to the developers.", e);
} }
} catch(Exception e){
LOGGER.error("AT States Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
} }
} catch (DataException e) {
LOGGER.warn(String.format("Repository issue trying to prune AT states: %s", e.getMessage()));
} catch (InterruptedException e) {
// Time to exit
} }
} }
} }

View File

@ -11,8 +11,6 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import static java.lang.Thread.MIN_PRIORITY;
public class AtStatesTrimmer implements Runnable { public class AtStatesTrimmer implements Runnable {
private static final Logger LOGGER = LogManager.getLogger(AtStatesTrimmer.class); private static final Logger LOGGER = LogManager.getLogger(AtStatesTrimmer.class);
@ -26,83 +24,66 @@ public class AtStatesTrimmer implements Runnable {
return; return;
} }
int trimStartHeight;
int maxLatestAtStatesHeight;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
trimStartHeight = repository.getATRepository().getAtTrimHeight(); int trimStartHeight = repository.getATRepository().getAtTrimHeight();
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository); int maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.discardChanges(); repository.discardChanges();
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight); repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges(); repository.saveChanges();
} catch (Exception e) {
LOGGER.error("AT States Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
return;
}
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
try (final Repository repository = RepositoryManager.getRepository()) { repository.discardChanges();
try {
repository.discardChanges();
Thread.sleep(Settings.getInstance().getAtStatesTrimInterval()); Thread.sleep(Settings.getInstance().getAtStatesTrimInterval());
BlockData chainTip = Controller.getInstance().getChainTip(); BlockData chainTip = Controller.getInstance().getChainTip();
if (chainTip == null || NTP.getTime() == null) if (chainTip == null || NTP.getTime() == null)
continue; continue;
// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages // Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (Synchronizer.getInstance().isSynchronizing()) if (Synchronizer.getInstance().isSynchronizing())
continue; continue;
long currentTrimmableTimestamp = NTP.getTime() - Settings.getInstance().getAtStatesMaxLifetime(); long currentTrimmableTimestamp = NTP.getTime() - Settings.getInstance().getAtStatesMaxLifetime();
// We want to keep AT states near the tip of our copy of blockchain so we can process/orphan nearby blocks // We want to keep AT states near the tip of our copy of blockchain so we can process/orphan nearby blocks
long chainTrimmableTimestamp = chainTip.getTimestamp() - Settings.getInstance().getAtStatesMaxLifetime(); long chainTrimmableTimestamp = chainTip.getTimestamp() - Settings.getInstance().getAtStatesMaxLifetime();
long upperTrimmableTimestamp = Math.min(currentTrimmableTimestamp, chainTrimmableTimestamp); long upperTrimmableTimestamp = Math.min(currentTrimmableTimestamp, chainTrimmableTimestamp);
int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp); int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
int upperBatchHeight = trimStartHeight + Settings.getInstance().getAtStatesTrimBatchSize(); int upperBatchHeight = trimStartHeight + Settings.getInstance().getAtStatesTrimBatchSize();
int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight); int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight);
if (trimStartHeight >= upperTrimHeight) if (trimStartHeight >= upperTrimHeight)
continue; continue;
int numAtStatesTrimmed = repository.getATRepository().trimAtStates(trimStartHeight, upperTrimHeight, Settings.getInstance().getAtStatesTrimLimit()); int numAtStatesTrimmed = repository.getATRepository().trimAtStates(trimStartHeight, upperTrimHeight, Settings.getInstance().getAtStatesTrimLimit());
repository.saveChanges(); repository.saveChanges();
if (numAtStatesTrimmed > 0) {
final int finalTrimStartHeight = trimStartHeight;
LOGGER.debug(() -> String.format("Trimmed %d AT state%s between blocks %d and %d",
numAtStatesTrimmed, (numAtStatesTrimmed != 1 ? "s" : ""),
finalTrimStartHeight, upperTrimHeight));
} else {
// Can we move onto next batch?
if (upperTrimmableHeight > upperBatchHeight) {
trimStartHeight = upperBatchHeight;
repository.getATRepository().setAtTrimHeight(trimStartHeight);
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges();
if (numAtStatesTrimmed > 0) {
final int finalTrimStartHeight = trimStartHeight; final int finalTrimStartHeight = trimStartHeight;
LOGGER.info(() -> String.format("Trimmed %d AT state%s between blocks %d and %d", LOGGER.debug(() -> String.format("Bumping AT state base trim height to %d", finalTrimStartHeight));
numAtStatesTrimmed, (numAtStatesTrimmed != 1 ? "s" : ""),
finalTrimStartHeight, upperTrimHeight));
} else {
// Can we move onto next batch?
if (upperTrimmableHeight > upperBatchHeight) {
trimStartHeight = upperBatchHeight;
repository.getATRepository().setAtTrimHeight(trimStartHeight);
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges();
final int finalTrimStartHeight = trimStartHeight;
LOGGER.info(() -> String.format("Bumping AT state base trim height to %d", finalTrimStartHeight));
}
} }
} catch (InterruptedException e) {
if(Controller.isStopping()) {
LOGGER.info("AT States Trimming Shutting Down");
}
else {
LOGGER.warn("AT States Trimming interrupted. Trying again. Report this error immediately to the developers.", e);
}
} catch (Exception e) {
LOGGER.warn("AT States Trimming stopped working. Trying again. Report this error immediately to the developers.", e);
} }
} catch (Exception e) {
LOGGER.error("AT States Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
} }
} catch (DataException e) {
LOGGER.warn(String.format("Repository issue trying to trim AT states: %s", e.getMessage()));
} catch (InterruptedException e) {
// Time to exit
} }
} }

View File

@ -15,13 +15,11 @@ import org.qortal.utils.NTP;
import java.io.IOException; import java.io.IOException;
import static java.lang.Thread.NORM_PRIORITY;
public class BlockArchiver implements Runnable { public class BlockArchiver implements Runnable {
private static final Logger LOGGER = LogManager.getLogger(BlockArchiver.class); private static final Logger LOGGER = LogManager.getLogger(BlockArchiver.class);
private static final long INITIAL_SLEEP_PERIOD = 15 * 60 * 1000L; // ms private static final long INITIAL_SLEEP_PERIOD = 5 * 60 * 1000L + 1234L; // ms
public void run() { public void run() {
Thread.currentThread().setName("Block archiver"); Thread.currentThread().setName("Block archiver");
@ -30,13 +28,11 @@ public class BlockArchiver implements Runnable {
return; return;
} }
int startHeight;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
// Don't even start building until initial rush has ended // Don't even start building until initial rush has ended
Thread.sleep(INITIAL_SLEEP_PERIOD); Thread.sleep(INITIAL_SLEEP_PERIOD);
startHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight(); int startHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight();
// Don't attempt to archive if we have no ATStatesHeightIndex, as it will be too slow // Don't attempt to archive if we have no ATStatesHeightIndex, as it will be too slow
boolean hasAtStatesHeightIndex = repository.getATRepository().hasAtStatesHeightIndex(); boolean hasAtStatesHeightIndex = repository.getATRepository().hasAtStatesHeightIndex();
@ -45,87 +41,77 @@ public class BlockArchiver implements Runnable {
repository.discardChanges(); repository.discardChanges();
return; return;
} }
} catch (Exception e) {
LOGGER.error("Block Archiving is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
return;
}
LOGGER.info("Starting block archiver from height {}...", startHeight); LOGGER.info("Starting block archiver from height {}...", startHeight);
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
try (final Repository repository = RepositoryManager.getRepository()) { repository.discardChanges();
try { Thread.sleep(Settings.getInstance().getArchiveInterval());
repository.discardChanges();
Thread.sleep(Settings.getInstance().getArchiveInterval()); BlockData chainTip = Controller.getInstance().getChainTip();
if (chainTip == null || NTP.getTime() == null) {
BlockData chainTip = Controller.getInstance().getChainTip(); continue;
if (chainTip == null || NTP.getTime() == null) {
continue;
}
// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (Synchronizer.getInstance().isSynchronizing()) {
continue;
}
// Don't attempt to archive if we're not synced yet
final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
continue;
}
// Build cache of blocks
try {
final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, repository);
BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
switch (result) {
case OK:
// Increment block archive height
startHeight += writer.getWrittenCount();
repository.getBlockArchiveRepository().setBlockArchiveHeight(startHeight);
repository.saveChanges();
break;
case STOPPING:
return;
// We've reached the limit of the blocks we can archive
// Sleep for a while to allow more to become available
case NOT_ENOUGH_BLOCKS:
// We didn't reach our file size target, so that must mean that we don't have enough blocks
// yet or something went wrong. Sleep for a while and then try again.
repository.discardChanges();
Thread.sleep(2 * 60 * 60 * 1000L); // 1 hour
break;
case BLOCK_NOT_FOUND:
// We tried to archive a block that didn't exist. This is a major failure and likely means
// that a bootstrap or re-sync is needed. Try again every minute until then.
LOGGER.info("Error: block not found when building archive. If this error persists, " +
"a bootstrap or re-sync may be needed.");
repository.discardChanges();
Thread.sleep(60 * 1000L); // 1 minute
break;
}
} catch (IOException | TransformationException e) {
LOGGER.info("Caught exception when creating block cache", e);
}
} catch (InterruptedException e) {
if (Controller.isStopping()) {
LOGGER.info("Block Archiving Shutting Down");
} else {
LOGGER.warn("Block Archiving interrupted. Trying again. Report this error immediately to the developers.", e);
}
} catch (Exception e) {
LOGGER.warn("Block Archiving stopped working. Trying again. Report this error immediately to the developers.", e);
} }
} catch(Exception e){
LOGGER.error("Block Archiving is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e); // Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (Synchronizer.getInstance().isSynchronizing()) {
continue;
}
// Don't attempt to archive if we're not synced yet
final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
continue;
}
// Build cache of blocks
try {
final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, repository);
BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
switch (result) {
case OK:
// Increment block archive height
startHeight += writer.getWrittenCount();
repository.getBlockArchiveRepository().setBlockArchiveHeight(startHeight);
repository.saveChanges();
break;
case STOPPING:
return;
// We've reached the limit of the blocks we can archive
// Sleep for a while to allow more to become available
case NOT_ENOUGH_BLOCKS:
// We didn't reach our file size target, so that must mean that we don't have enough blocks
// yet or something went wrong. Sleep for a while and then try again.
repository.discardChanges();
Thread.sleep(60 * 60 * 1000L); // 1 hour
break;
case BLOCK_NOT_FOUND:
// We tried to archive a block that didn't exist. This is a major failure and likely means
// that a bootstrap or re-sync is needed. Try again every minute until then.
LOGGER.info("Error: block not found when building archive. If this error persists, " +
"a bootstrap or re-sync may be needed.");
repository.discardChanges();
Thread.sleep( 60 * 1000L); // 1 minute
break;
}
} catch (IOException | TransformationException e) {
LOGGER.info("Caught exception when creating block cache", e);
}
} }
} catch (DataException e) {
LOGGER.info("Caught exception when creating block cache", e);
} catch (InterruptedException e) {
// Do nothing
} }
} }
} }

View File

@ -11,8 +11,6 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import static java.lang.Thread.NORM_PRIORITY;
public class BlockPruner implements Runnable { public class BlockPruner implements Runnable {
private static final Logger LOGGER = LogManager.getLogger(BlockPruner.class); private static final Logger LOGGER = LogManager.getLogger(BlockPruner.class);
@ -39,10 +37,8 @@ public class BlockPruner implements Runnable {
} }
} }
int pruneStartHeight;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
pruneStartHeight = repository.getBlockRepository().getBlockPruneHeight(); int pruneStartHeight = repository.getBlockRepository().getBlockPruneHeight();
// Don't attempt to prune if we have no ATStatesHeightIndex, as it will be too slow // Don't attempt to prune if we have no ATStatesHeightIndex, as it will be too slow
boolean hasAtStatesHeightIndex = repository.getATRepository().hasAtStatesHeightIndex(); boolean hasAtStatesHeightIndex = repository.getATRepository().hasAtStatesHeightIndex();
@ -50,90 +46,75 @@ public class BlockPruner implements Runnable {
LOGGER.info("Unable to start block pruner due to missing ATStatesHeightIndex. Bootstrapping is recommended."); LOGGER.info("Unable to start block pruner due to missing ATStatesHeightIndex. Bootstrapping is recommended.");
return; return;
} }
} catch (Exception e) {
LOGGER.error("Block Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
return;
}
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
repository.discardChanges();
try (final Repository repository = RepositoryManager.getRepository()) { Thread.sleep(Settings.getInstance().getBlockPruneInterval());
try { BlockData chainTip = Controller.getInstance().getChainTip();
repository.discardChanges(); if (chainTip == null || NTP.getTime() == null)
continue;
Thread.sleep(Settings.getInstance().getBlockPruneInterval()); // Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (Synchronizer.getInstance().isSynchronizing()) {
continue;
}
BlockData chainTip = Controller.getInstance().getChainTip(); // Don't attempt to prune if we're not synced yet
if (chainTip == null || NTP.getTime() == null) final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
continue; if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
continue;
}
// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages // Prune all blocks up until our latest minus pruneBlockLimit
if (Synchronizer.getInstance().isSynchronizing()) { final int ourLatestHeight = chainTip.getHeight();
continue; int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
}
// Don't attempt to prune if we're not synced yet // In archive mode we are only allowed to trim blocks that have already been archived
final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp(); if (archiveMode) {
if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) { upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
continue; }
}
// Prune all blocks up until our latest minus pruneBlockLimit int upperBatchHeight = pruneStartHeight + Settings.getInstance().getBlockPruneBatchSize();
final int ourLatestHeight = chainTip.getHeight(); int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
// In archive mode we are only allowed to trim blocks that have already been archived if (pruneStartHeight >= upperPruneHeight) {
if (archiveMode) { continue;
upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1; }
}
int upperBatchHeight = pruneStartHeight + Settings.getInstance().getBlockPruneBatchSize(); LOGGER.debug(String.format("Pruning blocks between %d and %d...", pruneStartHeight, upperPruneHeight));
int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
if (pruneStartHeight >= upperPruneHeight) { int numBlocksPruned = repository.getBlockRepository().pruneBlocks(pruneStartHeight, upperPruneHeight);
continue; repository.saveChanges();
}
LOGGER.info(String.format("Pruning blocks between %d and %d...", pruneStartHeight, upperPruneHeight)); if (numBlocksPruned > 0) {
LOGGER.debug(String.format("Pruned %d block%s between %d and %d",
int numBlocksPruned = repository.getBlockRepository().pruneBlocks(pruneStartHeight, upperPruneHeight); numBlocksPruned, (numBlocksPruned != 1 ? "s" : ""),
pruneStartHeight, upperPruneHeight));
} else {
final int nextPruneHeight = upperPruneHeight + 1;
repository.getBlockRepository().setBlockPruneHeight(nextPruneHeight);
repository.saveChanges(); repository.saveChanges();
LOGGER.debug(String.format("Bumping block base prune height to %d", pruneStartHeight));
if (numBlocksPruned > 0) { // Can we move onto next batch?
LOGGER.info(String.format("Pruned %d block%s between %d and %d", if (upperPrunableHeight > nextPruneHeight) {
numBlocksPruned, (numBlocksPruned != 1 ? "s" : ""), pruneStartHeight = nextPruneHeight;
pruneStartHeight, upperPruneHeight));
} else {
final int nextPruneHeight = upperPruneHeight + 1;
repository.getBlockRepository().setBlockPruneHeight(nextPruneHeight);
repository.saveChanges();
LOGGER.info(String.format("Bumping block base prune height to %d", pruneStartHeight));
// Can we move onto next batch?
if (upperPrunableHeight > nextPruneHeight) {
pruneStartHeight = nextPruneHeight;
}
else {
// We've pruned up to the upper prunable height
// Back off for a while to save CPU for syncing
repository.discardChanges();
Thread.sleep(10*60*1000L);
}
}
} catch (InterruptedException e) {
if(Controller.isStopping()) {
LOGGER.info("Block Pruning Shutting Down");
} }
else { else {
LOGGER.warn("Block Pruning interrupted. Trying again. Report this error immediately to the developers.", e); // We've pruned up to the upper prunable height
// Back off for a while to save CPU for syncing
repository.discardChanges();
Thread.sleep(10*60*1000L);
} }
} catch (Exception e) {
LOGGER.warn("Block Pruning stopped working. Trying again. Report this error immediately to the developers.", e);
} }
} catch(Exception e){
LOGGER.error("Block Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
} }
} catch (DataException e) {
LOGGER.warn(String.format("Repository issue trying to prune blocks: %s", e.getMessage()));
} catch (InterruptedException e) {
// Time to exit
} }
} }
} }

View File

@ -12,8 +12,6 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings; import org.qortal.settings.Settings;
import org.qortal.utils.NTP; import org.qortal.utils.NTP;
import static java.lang.Thread.NORM_PRIORITY;
public class OnlineAccountsSignaturesTrimmer implements Runnable { public class OnlineAccountsSignaturesTrimmer implements Runnable {
private static final Logger LOGGER = LogManager.getLogger(OnlineAccountsSignaturesTrimmer.class); private static final Logger LOGGER = LogManager.getLogger(OnlineAccountsSignaturesTrimmer.class);
@ -28,77 +26,61 @@ public class OnlineAccountsSignaturesTrimmer implements Runnable {
return; return;
} }
int trimStartHeight;
try (final Repository repository = RepositoryManager.getRepository()) { try (final Repository repository = RepositoryManager.getRepository()) {
// Don't even start trimming until initial rush has ended // Don't even start trimming until initial rush has ended
Thread.sleep(INITIAL_SLEEP_PERIOD); Thread.sleep(INITIAL_SLEEP_PERIOD);
trimStartHeight = repository.getBlockRepository().getOnlineAccountsSignaturesTrimHeight(); int trimStartHeight = repository.getBlockRepository().getOnlineAccountsSignaturesTrimHeight();
} catch (Exception e) {
LOGGER.error("Online Accounts Signatures Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
return;
}
while (!Controller.isStopping()) { while (!Controller.isStopping()) {
try (final Repository repository = RepositoryManager.getRepository()) { repository.discardChanges();
try { Thread.sleep(Settings.getInstance().getOnlineSignaturesTrimInterval());
repository.discardChanges();
Thread.sleep(Settings.getInstance().getOnlineSignaturesTrimInterval()); BlockData chainTip = Controller.getInstance().getChainTip();
if (chainTip == null || NTP.getTime() == null)
continue;
BlockData chainTip = Controller.getInstance().getChainTip(); // Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (chainTip == null || NTP.getTime() == null) if (Synchronizer.getInstance().isSynchronizing())
continue; continue;
// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages // Trim blockchain by removing 'old' online accounts signatures
if (Synchronizer.getInstance().isSynchronizing()) long upperTrimmableTimestamp = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMaxLifetime();
continue; int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
// Trim blockchain by removing 'old' online accounts signatures int upperBatchHeight = trimStartHeight + Settings.getInstance().getOnlineSignaturesTrimBatchSize();
long upperTrimmableTimestamp = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMaxLifetime(); int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight);
int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
int upperBatchHeight = trimStartHeight + Settings.getInstance().getOnlineSignaturesTrimBatchSize(); if (trimStartHeight >= upperTrimHeight)
int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight); continue;
if (trimStartHeight >= upperTrimHeight) int numSigsTrimmed = repository.getBlockRepository().trimOldOnlineAccountsSignatures(trimStartHeight, upperTrimHeight);
continue; repository.saveChanges();
int numSigsTrimmed = repository.getBlockRepository().trimOldOnlineAccountsSignatures(trimStartHeight, upperTrimHeight); if (numSigsTrimmed > 0) {
repository.saveChanges(); final int finalTrimStartHeight = trimStartHeight;
LOGGER.debug(() -> String.format("Trimmed %d online accounts signature%s between blocks %d and %d",
numSigsTrimmed, (numSigsTrimmed != 1 ? "s" : ""),
finalTrimStartHeight, upperTrimHeight));
} else {
// Can we move onto next batch?
if (upperTrimmableHeight > upperBatchHeight) {
trimStartHeight = upperBatchHeight;
repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(trimStartHeight);
repository.saveChanges();
if (numSigsTrimmed > 0) {
final int finalTrimStartHeight = trimStartHeight; final int finalTrimStartHeight = trimStartHeight;
LOGGER.info(() -> String.format("Trimmed %d online accounts signature%s between blocks %d and %d", LOGGER.debug(() -> String.format("Bumping online accounts signatures base trim height to %d", finalTrimStartHeight));
numSigsTrimmed, (numSigsTrimmed != 1 ? "s" : ""),
finalTrimStartHeight, upperTrimHeight));
} else {
// Can we move onto next batch?
if (upperTrimmableHeight > upperBatchHeight) {
trimStartHeight = upperBatchHeight;
repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(trimStartHeight);
repository.saveChanges();
final int finalTrimStartHeight = trimStartHeight;
LOGGER.info(() -> String.format("Bumping online accounts signatures base trim height to %d", finalTrimStartHeight));
}
} }
} catch (InterruptedException e) {
if(Controller.isStopping()) {
LOGGER.info("Online Accounts Signatures Trimming Shutting Down");
}
else {
LOGGER.warn("Online Accounts Signatures Trimming interrupted. Trying again. Report this error immediately to the developers.", e);
}
} catch (Exception e) {
LOGGER.warn("Online Accounts Signatures Trimming stopped working. Trying again. Report this error immediately to the developers.", e);
} }
} catch (Exception e) {
LOGGER.error("Online Accounts Signatures Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
} }
} catch (DataException e) {
LOGGER.warn(String.format("Repository issue trying to trim online accounts signatures: %s", e.getMessage()));
} catch (InterruptedException e) {
// Time to exit
} }
} }
} }

View File

@ -40,7 +40,7 @@ public class PruneManager {
} }
public void start() { public void start() {
this.executorService = Executors.newCachedThreadPool(new DaemonThreadFactory(Settings.getInstance().getPruningThreadPriority())); this.executorService = Executors.newCachedThreadPool(new DaemonThreadFactory());
if (Settings.getInstance().isTopOnly()) { if (Settings.getInstance().isTopOnly()) {
// Top-only-sync // Top-only-sync

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -528,7 +527,7 @@ public class BitcoinACCTv1TradeBot implements AcctTradeBot {
// P2SH-A funding confirmed // P2SH-A funding confirmed
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = BitcoinACCTv1.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -894,7 +893,7 @@ public class BitcoinACCTv1TradeBot implements AcctTradeBot {
// Redeem P2SH-B using secret-B // Redeem P2SH-B using secret-B
Coin redeemAmount = Coin.valueOf(P2SH_B_OUTPUT_AMOUNT); // An actual amount to avoid dust filter, remaining used as fees. The real funds are in P2SH-A. Coin redeemAmount = Coin.valueOf(P2SH_B_OUTPUT_AMOUNT); // An actual amount to avoid dust filter, remaining used as fees. The real funds are in P2SH-A.
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressB, false); List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressB);
byte[] receivingAccountInfo = tradeBotData.getReceivingAccountInfo(); byte[] receivingAccountInfo = tradeBotData.getReceivingAccountInfo();
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoin.getNetworkParameters(), redeemAmount, redeemKey,
@ -1064,7 +1063,7 @@ public class BitcoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount - P2SH_B_OUTPUT_AMOUNT); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount - P2SH_B_OUTPUT_AMOUNT);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -1136,7 +1135,7 @@ public class BitcoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(P2SH_B_OUTPUT_AMOUNT); // An actual amount to avoid dust filter, remaining used as fees. Coin refundAmount = Coin.valueOf(P2SH_B_OUTPUT_AMOUNT); // An actual amount to avoid dust filter, remaining used as fees.
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressB, false); List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressB);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = bitcoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = bitcoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());
@ -1202,7 +1201,7 @@ public class BitcoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount - P2SH_B_OUTPUT_AMOUNT); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount - P2SH_B_OUTPUT_AMOUNT);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = bitcoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = bitcoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -7,9 +7,7 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.controller.tradebot.TradeStates.State;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
import org.qortal.data.at.ATData; import org.qortal.data.at.ATData;
@ -32,8 +30,12 @@ import org.qortal.utils.NTP;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
* <p> * <p>
@ -48,6 +50,45 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
private static final Logger LOGGER = LogManager.getLogger(BitcoinACCTv3TradeBot.class); private static final Logger LOGGER = LogManager.getLogger(BitcoinACCTv3TradeBot.class);
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
/** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */ /** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */
private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms
@ -272,7 +313,7 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = BitcoinACCTv3.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -752,7 +793,7 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(bitcoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -816,7 +857,7 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = bitcoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = bitcoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = bitcoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -31,9 +30,11 @@ import org.qortal.utils.NTP;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.qortal.controller.tradebot.TradeStates.State; import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
@ -49,6 +50,45 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
private static final Logger LOGGER = LogManager.getLogger(DigibyteACCTv3TradeBot.class); private static final Logger LOGGER = LogManager.getLogger(DigibyteACCTv3TradeBot.class);
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
/** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */ /** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */
private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms
@ -273,7 +313,7 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = DigibyteACCTv3.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -753,7 +793,7 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = digibyte.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = digibyte.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(digibyte.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(digibyte.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -817,7 +857,7 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = digibyte.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = digibyte.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = digibyte.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = digibyte.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -314,7 +313,7 @@ public class DogecoinACCTv1TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = DogecoinACCTv1.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -794,7 +793,7 @@ public class DogecoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(dogecoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(dogecoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -858,7 +857,7 @@ public class DogecoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = dogecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = dogecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -31,9 +30,11 @@ import org.qortal.utils.NTP;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.qortal.controller.tradebot.TradeStates.State; import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
@ -49,6 +50,45 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
private static final Logger LOGGER = LogManager.getLogger(DogecoinACCTv3TradeBot.class); private static final Logger LOGGER = LogManager.getLogger(DogecoinACCTv3TradeBot.class);
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
/** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */ /** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */
private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms
@ -273,7 +313,7 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = DogecoinACCTv3.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -753,7 +793,7 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(dogecoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(dogecoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -817,7 +857,7 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = dogecoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = dogecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = dogecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -313,7 +312,7 @@ public class LitecoinACCTv1TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = LitecoinACCTv1.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -757,7 +756,7 @@ public class LitecoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(litecoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(litecoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -821,7 +820,7 @@ public class LitecoinACCTv1TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = litecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = litecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -31,9 +30,12 @@ import org.qortal.utils.NTP;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.qortal.controller.tradebot.TradeStates.State; import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
* <p> * <p>
@ -48,6 +50,45 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
private static final Logger LOGGER = LogManager.getLogger(LitecoinACCTv3TradeBot.class); private static final Logger LOGGER = LogManager.getLogger(LitecoinACCTv3TradeBot.class);
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
/** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */ /** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */
private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms
@ -272,7 +313,7 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = LitecoinACCTv3.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -752,7 +793,7 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(litecoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(litecoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -816,7 +857,7 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = litecoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = litecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = litecoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -9,7 +9,6 @@ import org.bitcoinj.core.Coin;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -33,9 +32,11 @@ import org.qortal.utils.NTP;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.qortal.controller.tradebot.TradeStates.State; import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
@ -51,6 +52,45 @@ public class PirateChainACCTv3TradeBot implements AcctTradeBot {
private static final Logger LOGGER = LogManager.getLogger(PirateChainACCTv3TradeBot.class); private static final Logger LOGGER = LogManager.getLogger(PirateChainACCTv3TradeBot.class);
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
/** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */ /** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */
private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms
@ -277,7 +317,7 @@ public class PirateChainACCTv3TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKey(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = PirateChainACCTv3.buildOfferMessage(tradeBotData.getTradeForeignPublicKey(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);

View File

@ -7,7 +7,6 @@ import org.bitcoinj.script.Script.ScriptType;
import org.qortal.account.PrivateKeyAccount; import org.qortal.account.PrivateKeyAccount;
import org.qortal.account.PublicKeyAccount; import org.qortal.account.PublicKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.asset.Asset; import org.qortal.asset.Asset;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -31,9 +30,11 @@ import org.qortal.utils.NTP;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.qortal.controller.tradebot.TradeStates.State; import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
@ -49,6 +50,45 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
private static final Logger LOGGER = LogManager.getLogger(RavencoinACCTv3TradeBot.class); private static final Logger LOGGER = LogManager.getLogger(RavencoinACCTv3TradeBot.class);
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
/** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */ /** Maximum time Bob waits for his AT creation transaction to be confirmed into a block. (milliseconds) */
private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms private static final long MAX_AT_CONFIRMATION_PERIOD = 24 * 60 * 60 * 1000L; // ms
@ -273,7 +313,7 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
} }
// Attempt to send MESSAGE to Bob's Qortal trade address // Attempt to send MESSAGE to Bob's Qortal trade address
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA()); byte[] messageData = RavencoinACCTv3.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress; String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData); boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
@ -753,7 +793,7 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED: { case FUNDED: {
Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin redeemAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey redeemKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = ravencoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = ravencoin.getUnspentOutputs(p2shAddressA);
Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(ravencoin.getNetworkParameters(), redeemAmount, redeemKey, Transaction p2shRedeemTransaction = BitcoinyHTLC.buildRedeemTransaction(ravencoin.getNetworkParameters(), redeemAmount, redeemKey,
fundingOutputs, redeemScriptA, secretA, receivingAccountInfo); fundingOutputs, redeemScriptA, secretA, receivingAccountInfo);
@ -817,7 +857,7 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
case FUNDED:{ case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount); Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey()); ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = ravencoin.getUnspentOutputs(p2shAddressA, false); List<TransactionOutput> fundingOutputs = ravencoin.getUnspentOutputs(p2shAddressA);
// Determine receive address for refund // Determine receive address for refund
String receiveAddress = ravencoin.getUnusedReceiveAddress(tradeBotData.getForeignKey()); String receiveAddress = ravencoin.getUnusedReceiveAddress(tradeBotData.getForeignKey());

View File

@ -8,7 +8,6 @@ import org.qortal.account.PrivateKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest; import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.controller.Controller; import org.qortal.controller.Controller;
import org.qortal.controller.Synchronizer; import org.qortal.controller.Synchronizer;
import org.qortal.controller.arbitrary.PeerMessage;
import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult; import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult;
import org.qortal.crosschain.*; import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto; import org.qortal.crypto.Crypto;
@ -38,12 +37,7 @@ import org.qortal.utils.NTP;
import java.awt.TrayIcon.MessageType; import java.awt.TrayIcon.MessageType;
import java.security.SecureRandom; import java.security.SecureRandom;
import java.util.*; import java.util.*;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.function.Supplier; import java.util.function.Supplier;
import java.util.stream.Collectors;
/** /**
* Performing cross-chain trading steps on behalf of user. * Performing cross-chain trading steps on behalf of user.
@ -124,9 +118,6 @@ public class TradeBot implements Listener {
private Map<String, Long> validTrades = new HashMap<>(); private Map<String, Long> validTrades = new HashMap<>();
private TradeBot() { private TradeBot() {
tradePresenceMessageScheduler.scheduleAtFixedRate( this::processTradePresencesMessages, 60, 1, TimeUnit.SECONDS);
EventBus.INSTANCE.addListener(event -> TradeBot.getInstance().listen(event)); EventBus.INSTANCE.addListener(event -> TradeBot.getInstance().listen(event));
} }
@ -224,41 +215,6 @@ public class TradeBot implements Listener {
return acctTradeBot.startResponse(repository, atData, acct, crossChainTradeData, foreignKey, receivingAddress); return acctTradeBot.startResponse(repository, atData, acct, crossChainTradeData, foreignKey, receivingAddress);
} }
/**
* Creates a trade-bot entries from the 'Alice' viewpoint,
* i.e. matching foreign blockchain currency to existing QORT offers.
* <p>
* Requires chosen trade offers from Bob, passed by <tt>crossChainTradeData</tt>
* and access to a foreign blockchain wallet via <tt>foreignKey</tt>.
* <p>
* @param repository
* @param crossChainTradeDataList chosen trade OFFERs that Alice wants to match
* @param receiveAddress Alice's Qortal address to receive her QORT
* @param foreignKey foreign blockchain wallet key
* @param bitcoiny
* @throws DataException
*/
public ResponseResult startResponseMultiple(
Repository repository,
ACCT acct,
List<CrossChainTradeData> crossChainTradeDataList,
String receiveAddress,
String foreignKey,
Bitcoiny bitcoiny) throws DataException {
AcctTradeBot acctTradeBot = findTradeBotForAcct(acct);
if (acctTradeBot == null) {
LOGGER.debug(() -> String.format("Couldn't find ACCT trade-bot for %s", acct.getBlockchain()));
return ResponseResult.NETWORK_ISSUE;
}
for( CrossChainTradeData tradeData : crossChainTradeDataList) {
// Check Alice doesn't already have an existing, on-going trade-bot entry for this AT.
if (repository.getCrossChainRepository().existsTradeWithAtExcludingStates(tradeData.qortalAtAddress, acctTradeBot.getEndStates()))
return ResponseResult.TRADE_ALREADY_EXISTS;
}
return TradeBotUtils.startResponseMultiple(repository, acct, crossChainTradeDataList, receiveAddress, foreignKey, bitcoiny);
}
public boolean deleteEntry(Repository repository, byte[] tradePrivateKey) throws DataException { public boolean deleteEntry(Repository repository, byte[] tradePrivateKey) throws DataException {
TradeBotData tradeBotData = repository.getCrossChainRepository().getTradeBotData(tradePrivateKey); TradeBotData tradeBotData = repository.getCrossChainRepository().getTradeBotData(tradePrivateKey);
if (tradeBotData == null) if (tradeBotData == null)
@ -560,139 +516,77 @@ public class TradeBot implements Listener {
} }
} }
// List to collect messages
private final List<PeerMessage> tradePresenceMessageList = new ArrayList<>();
// Lock to synchronize access to the list
private final Object tradePresenceMessageLock = new Object();
// Scheduled executor service to process messages every second
private final ScheduledExecutorService tradePresenceMessageScheduler = Executors.newScheduledThreadPool(1);
public void onTradePresencesMessage(Peer peer, Message message) { public void onTradePresencesMessage(Peer peer, Message message) {
TradePresencesMessage tradePresencesMessage = (TradePresencesMessage) message;
synchronized (tradePresenceMessageLock) { List<TradePresenceData> peersTradePresences = tradePresencesMessage.getTradePresences();
tradePresenceMessageList.add(new PeerMessage(peer, message));
}
}
public void processTradePresencesMessages() { long now = NTP.getTime();
// Timestamps before this are too far into the past
long pastThreshold = now;
// Timestamps after this are too far into the future
long futureThreshold = now + PRESENCE_LIFETIME;
try { Map<ByteArray, Supplier<ACCT>> acctSuppliersByCodeHash = SupportedBlockchain.getAcctMap();
List<PeerMessage> messagesToProcess;
synchronized (tradePresenceMessageLock) {
messagesToProcess = new ArrayList<>(tradePresenceMessageList);
tradePresenceMessageList.clear();
}
if( messagesToProcess.isEmpty() ) return; int newCount = 0;
Map<Peer, List<TradePresenceData>> tradePresencesByPeer = new HashMap<>(messagesToProcess.size()); try (final Repository repository = RepositoryManager.getRepository()) {
for (TradePresenceData peersTradePresence : peersTradePresences) {
long timestamp = peersTradePresence.getTimestamp();
// map all trade presences from the messages to their peer // Ignore if timestamp is out of bounds
for( PeerMessage peerMessage : messagesToProcess ) { if (timestamp < pastThreshold || timestamp > futureThreshold) {
TradePresencesMessage tradePresencesMessage = (TradePresencesMessage) peerMessage.getMessage(); if (timestamp < pastThreshold)
LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too old vs {}",
List<TradePresenceData> peersTradePresences = tradePresencesMessage.getTradePresences(); peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
);
tradePresencesByPeer.put(peerMessage.getPeer(), peersTradePresences); else
} LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too new vs {}",
peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
long now = NTP.getTime();
// Timestamps before this are too far into the past
long pastThreshold = now;
// Timestamps after this are too far into the future
long futureThreshold = now + PRESENCE_LIFETIME;
Map<ByteArray, Supplier<ACCT>> acctSuppliersByCodeHash = SupportedBlockchain.getAcctMap();
int newCount = 0;
Map<String, List<Peer>> peersByAtAddress = new HashMap<>(tradePresencesByPeer.size());
Map<String, TradePresenceData> tradePresenceByAtAddress = new HashMap<>(tradePresencesByPeer.size());
// for each batch of trade presence data from a peer, validate and populate the maps declared above
for ( Map.Entry<Peer, List<TradePresenceData>> entry: tradePresencesByPeer.entrySet()) {
Peer peer = entry.getKey();
for( TradePresenceData peersTradePresence : entry.getValue() ) {
// TradePresenceData peersTradePresence
long timestamp = peersTradePresence.getTimestamp();
// Ignore if timestamp is out of bounds
if (timestamp < pastThreshold || timestamp > futureThreshold) {
if (timestamp < pastThreshold)
LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too old vs {}",
peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
);
else
LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too new vs {}",
peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
);
continue;
}
ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
// Ignore if we've previously verified this timestamp+publickey combo or sent timestamp is older
TradePresenceData existingTradeData = this.safeAllTradePresencesByPubkey.get(pubkeyByteArray);
if (existingTradeData != null && timestamp <= existingTradeData.getTimestamp()) {
if (timestamp == existingTradeData.getTimestamp())
LOGGER.trace("Ignoring trade presence {} from peer {} as we have verified timestamp {} before",
peersTradePresence.getAtAddress(), peer, timestamp
);
else
LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is older than latest {}",
peersTradePresence.getAtAddress(), peer, timestamp, existingTradeData.getTimestamp()
);
continue;
}
// Check timestamp signature
byte[] timestampSignature = peersTradePresence.getSignature();
byte[] timestampBytes = Longs.toByteArray(timestamp);
byte[] publicKey = peersTradePresence.getPublicKey();
if (!Crypto.verify(publicKey, timestampSignature, timestampBytes)) {
LOGGER.trace("Ignoring trade presence {} from peer {} as signature failed to verify",
peersTradePresence.getAtAddress(), peer
); );
continue; continue;
}
peersByAtAddress.computeIfAbsent(peersTradePresence.getAtAddress(), address -> new ArrayList<>()).add(peer);
tradePresenceByAtAddress.put(peersTradePresence.getAtAddress(), peersTradePresence);
} }
}
if( tradePresenceByAtAddress.isEmpty() ) return; ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
List<ATData> atDataList; // Ignore if we've previously verified this timestamp+publickey combo or sent timestamp is older
try (final Repository repository = RepositoryManager.getRepository()) { TradePresenceData existingTradeData = this.safeAllTradePresencesByPubkey.get(pubkeyByteArray);
atDataList = repository.getATRepository().fromATAddresses( new ArrayList<>(tradePresenceByAtAddress.keySet()) ); if (existingTradeData != null && timestamp <= existingTradeData.getTimestamp()) {
} catch (DataException e) { if (timestamp == existingTradeData.getTimestamp())
LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e); LOGGER.trace("Ignoring trade presence {} from peer {} as we have verified timestamp {} before",
return; peersTradePresence.getAtAddress(), peer, timestamp
}
Map<String, Supplier<ACCT>> supplierByAtAddress = new HashMap<>(atDataList.size());
List<ATData> validatedAtDataList = new ArrayList<>(atDataList.size());
// for each trade
for( ATData atData : atDataList ) {
TradePresenceData peersTradePresence = tradePresenceByAtAddress.get(atData.getATAddress());
if (atData == null || atData.getIsFrozen() || atData.getIsFinished()) {
if (atData == null)
LOGGER.trace("Ignoring trade presence {} from peer as AT doesn't exist",
peersTradePresence.getAtAddress()
); );
else else
LOGGER.trace("Ignoring trade presence {} from peer as AT is frozen or finished", LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is older than latest {}",
peersTradePresence.getAtAddress() peersTradePresence.getAtAddress(), peer, timestamp, existingTradeData.getTimestamp()
);
continue;
}
// Check timestamp signature
byte[] timestampSignature = peersTradePresence.getSignature();
byte[] timestampBytes = Longs.toByteArray(timestamp);
byte[] publicKey = peersTradePresence.getPublicKey();
if (!Crypto.verify(publicKey, timestampSignature, timestampBytes)) {
LOGGER.trace("Ignoring trade presence {} from peer {} as signature failed to verify",
peersTradePresence.getAtAddress(), peer
);
continue;
}
ATData atData = repository.getATRepository().fromATAddress(peersTradePresence.getAtAddress());
if (atData == null || atData.getIsFrozen() || atData.getIsFinished()) {
if (atData == null)
LOGGER.trace("Ignoring trade presence {} from peer {} as AT doesn't exist",
peersTradePresence.getAtAddress(), peer
);
else
LOGGER.trace("Ignoring trade presence {} from peer {} as AT is frozen or finished",
peersTradePresence.getAtAddress(), peer
); );
continue; continue;
@ -701,87 +595,51 @@ public class TradeBot implements Listener {
ByteArray atCodeHash = ByteArray.wrap(atData.getCodeHash()); ByteArray atCodeHash = ByteArray.wrap(atData.getCodeHash());
Supplier<ACCT> acctSupplier = acctSuppliersByCodeHash.get(atCodeHash); Supplier<ACCT> acctSupplier = acctSuppliersByCodeHash.get(atCodeHash);
if (acctSupplier == null) { if (acctSupplier == null) {
LOGGER.trace("Ignoring trade presence {} from peer as AT isn't a known ACCT?", LOGGER.trace("Ignoring trade presence {} from peer {} as AT isn't a known ACCT?",
peersTradePresence.getAtAddress() peersTradePresence.getAtAddress(), peer
); );
continue; continue;
} }
validatedAtDataList.add(atData);
}
// populated data for each trade CrossChainTradeData tradeData = acctSupplier.get().populateTradeData(repository, atData);
List<CrossChainTradeData> crossChainTradeDataList; if (tradeData == null) {
LOGGER.trace("Ignoring trade presence {} from peer {} as trade data not found?",
// validated trade data grouped by code (cross chain coin) peersTradePresence.getAtAddress(), peer
Map<ByteArray, List<ATData>> atDataByCodeHash
= validatedAtDataList.stream().collect(
Collectors.groupingBy(data -> ByteArray.wrap(data.getCodeHash())));
try (final Repository repository = RepositoryManager.getRepository()) {
crossChainTradeDataList = new ArrayList<>();
// for each code (cross chain coin), get each trade, then populate trade data
for( Map.Entry<ByteArray, List<ATData>> entry : atDataByCodeHash.entrySet() ) {
Supplier<ACCT> acctSupplier = acctSuppliersByCodeHash.get(entry.getKey());
crossChainTradeDataList.addAll(
acctSupplier.get().populateTradeDataList(
repository,
entry.getValue()
)
.stream().filter( data -> data != null )
.collect(Collectors.toList())
);
}
} catch (DataException e) {
LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e);
return;
}
// for each populated trade data, validate and fire event
for( CrossChainTradeData tradeData : crossChainTradeDataList ) {
List<Peer> peers = peersByAtAddress.get(tradeData.qortalAtAddress);
for( Peer peer : peers ) {
TradePresenceData peersTradePresence = tradePresenceByAtAddress.get(tradeData.qortalAtAddress);
// Convert signer's public key to address form
String signerAddress = peersTradePresence.getTradeAddress();
// Signer's public key (in address form) must match Bob's / Alice's trade public key (in address form)
if (!signerAddress.equals(tradeData.qortalCreatorTradeAddress) && !signerAddress.equals(tradeData.qortalPartnerAddress)) {
LOGGER.trace("Ignoring trade presence {} from peer {} as signer isn't Alice or Bob?",
peersTradePresence.getAtAddress(), peer
);
continue;
}
ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
// This is new to us
this.allTradePresencesByPubkey.put(pubkeyByteArray, peersTradePresence);
++newCount;
LOGGER.trace("Added trade presence {} from peer {} with timestamp {}",
peersTradePresence.getAtAddress(), peer, tradeData.creationTimestamp
); );
EventBus.INSTANCE.notify(new TradePresenceEvent(peersTradePresence)); continue;
} }
}
if (newCount > 0) { // Convert signer's public key to address form
LOGGER.info("New trade presences: {}, all trade presences: {}", newCount, allTradePresencesByPubkey.size()); String signerAddress = peersTradePresence.getTradeAddress();
rebuildSafeAllTradePresences();
// Signer's public key (in address form) must match Bob's / Alice's trade public key (in address form)
if (!signerAddress.equals(tradeData.qortalCreatorTradeAddress) && !signerAddress.equals(tradeData.qortalPartnerAddress)) {
LOGGER.trace("Ignoring trade presence {} from peer {} as signer isn't Alice or Bob?",
peersTradePresence.getAtAddress(), peer
);
continue;
}
// This is new to us
this.allTradePresencesByPubkey.put(pubkeyByteArray, peersTradePresence);
++newCount;
LOGGER.trace("Added trade presence {} from peer {} with timestamp {}",
peersTradePresence.getAtAddress(), peer, timestamp
);
EventBus.INSTANCE.notify(new TradePresenceEvent(peersTradePresence));
} }
} catch (Exception e) { } catch (DataException e) {
LOGGER.error(e.getMessage(), e); LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e);
}
if (newCount > 0) {
LOGGER.debug("New trade presences: {}, all trade presences: {}", newCount, allTradePresencesByPubkey.size());
rebuildSafeAllTradePresences();
} }
} }

View File

@ -1,217 +0,0 @@
package org.qortal.controller.tradebot;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.bitcoinj.core.Transaction;
import org.qortal.account.PrivateKeyAccount;
import org.qortal.api.resource.CrossChainUtils;
import org.qortal.crosschain.ACCT;
import org.qortal.crosschain.Bitcoiny;
import org.qortal.crosschain.BitcoinyHTLC;
import org.qortal.crosschain.ForeignBlockchainException;
import org.qortal.crypto.Crypto;
import org.qortal.data.crosschain.CrossChainTradeData;
import org.qortal.data.crosschain.TradeBotData;
import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.MessageTransaction;
import org.qortal.utils.Base58;
import org.qortal.utils.NTP;
import org.qortal.transaction.Transaction.ValidationResult;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.qortal.controller.tradebot.TradeStates.State;
public class TradeBotUtils {
private static final Logger LOGGER = LogManager.getLogger(TradeBotUtils.class);
/**
* Creates trade-bot entries from the 'Alice' viewpoint, i.e. matching Bitcoiny coin to existing offers.
* <p>
* Requires chosen trade offers from Bob, passed by <tt>crossChainTradeData</tt>
* and access to a Blockchain wallet via <tt>foreignKey</tt>.
* <p>
* The <tt>crossChainTradeData</tt> contains the current trade offers state
* as extracted from the AT's data segment.
* <p>
* Access to a funded wallet is via a Blockchain BIP32 hierarchical deterministic key,
* passed via <tt>foreignKey</tt>.
* <b>This key will be stored in your node's database</b>
* to allow trade-bot to create/fund the necessary P2SH transactions!
* However, due to the nature of BIP32 keys, it is possible to give the trade-bot
* only a subset of wallet access (see BIP32 for more details).
* <p>
* As an example, the foreignKey can be extract from a <i>legacy, password-less</i>
* Electrum wallet by going to the console tab and entering:<br>
* <tt>wallet.keystore.xprv</tt><br>
* which should result in a base58 string starting with either 'xprv' (for Blockchain main-net)
* or 'tprv' for (Blockchain test-net).
* <p>
* It is envisaged that the value in <tt>foreignKey</tt> will actually come from a Qortal-UI-managed wallet.
* <p>
* If sufficient funds are available, <b>this method will actually fund the P2SH-A</b>
* with the Blockchain amount expected by 'Bob'.
* <p>
* If the Blockchain transaction is successfully broadcast to the network then
* we also send a MESSAGE to Bob's trade-bot to let them know; one message for each trade.
* <p>
* The trade-bot entries are saved to the repository and the cross-chain trading process commences.
* <p>
*
* @param repository for backing up the trade bot data
* @param crossChainTradeDataList chosen trade OFFERs that Alice wants to match
* @param receiveAddress Alice's Qortal address
* @param foreignKey funded wallet xprv in base58
* @param bitcoiny the bitcoiny chain to match the sell offer with
* @return true if P2SH-A funding transaction successfully broadcast to Blockchain network, false otherwise
* @throws DataException
*/
public static AcctTradeBot.ResponseResult startResponseMultiple(
Repository repository,
ACCT acct,
List<CrossChainTradeData> crossChainTradeDataList,
String receiveAddress,
String foreignKey,
Bitcoiny bitcoiny) throws DataException {
// Check we have enough funds via foreignKey to fund P2SH to cover expectedForeignAmount
long now = NTP.getTime();
long p2shFee;
try {
p2shFee = bitcoiny.getP2shFee(now);
} catch (ForeignBlockchainException e) {
LOGGER.debug("Couldn't estimate blockchain transaction fees?");
return AcctTradeBot.ResponseResult.NETWORK_ISSUE;
}
Map<String, Long> valueByP2shAddress = new HashMap<>(crossChainTradeDataList.size());
class DataCombiner{
CrossChainTradeData crossChainTradeData;
TradeBotData tradeBotData;
String p2shAddress;
public DataCombiner(CrossChainTradeData crossChainTradeData, TradeBotData tradeBotData, String p2shAddress) {
this.crossChainTradeData = crossChainTradeData;
this.tradeBotData = tradeBotData;
this.p2shAddress = p2shAddress;
}
}
List<DataCombiner> dataToProcess = new ArrayList<>();
for(CrossChainTradeData crossChainTradeData : crossChainTradeDataList) {
byte[] tradePrivateKey = TradeBot.generateTradePrivateKey();
byte[] secretA = TradeBot.generateSecret();
byte[] hashOfSecretA = Crypto.hash160(secretA);
byte[] tradeNativePublicKey = TradeBot.deriveTradeNativePublicKey(tradePrivateKey);
byte[] tradeNativePublicKeyHash = Crypto.hash160(tradeNativePublicKey);
String tradeNativeAddress = Crypto.toAddress(tradeNativePublicKey);
byte[] tradeForeignPublicKey = TradeBot.deriveTradeForeignPublicKey(tradePrivateKey);
byte[] tradeForeignPublicKeyHash = Crypto.hash160(tradeForeignPublicKey);
// We need to generate lockTime-A: add tradeTimeout to now
int lockTimeA = (crossChainTradeData.tradeTimeout * 60) + (int) (now / 1000L);
byte[] receivingPublicKeyHash = Base58.decode(receiveAddress); // Actually the whole address, not just PKH
TradeBotData tradeBotData = new TradeBotData(tradePrivateKey, acct.getClass().getSimpleName(),
State.ALICE_WAITING_FOR_AT_LOCK.name(), State.ALICE_WAITING_FOR_AT_LOCK.value,
receiveAddress,
crossChainTradeData.qortalAtAddress,
now,
crossChainTradeData.qortAmount,
tradeNativePublicKey, tradeNativePublicKeyHash, tradeNativeAddress,
secretA, hashOfSecretA,
crossChainTradeData.foreignBlockchain,
tradeForeignPublicKey, tradeForeignPublicKeyHash,
crossChainTradeData.expectedForeignAmount,
foreignKey, null, lockTimeA, receivingPublicKeyHash);
// Attempt to backup the trade bot data
// Include tradeBotData as an additional parameter, since it's not in the repository yet
TradeBot.backupTradeBotData(repository, Arrays.asList(tradeBotData));
// Fee for redeem/refund is subtracted from P2SH-A balance.
// Do not include fee for funding transaction as this is covered by buildSpend()
long amountA = crossChainTradeData.expectedForeignAmount + p2shFee /*redeeming/refunding P2SH-A*/;
// P2SH-A to be funded
byte[] redeemScriptBytes = BitcoinyHTLC.buildScript(tradeForeignPublicKeyHash, lockTimeA, crossChainTradeData.creatorForeignPKH, hashOfSecretA);
String p2shAddress = bitcoiny.deriveP2shAddress(redeemScriptBytes);
valueByP2shAddress.put(p2shAddress, amountA);
dataToProcess.add(new DataCombiner(crossChainTradeData, tradeBotData, p2shAddress));
}
// Build transaction for funding P2SH-A
Transaction p2shFundingTransaction = bitcoiny.buildSpendMultiple(foreignKey, valueByP2shAddress, null);
if (p2shFundingTransaction == null) {
LOGGER.debug("Unable to build P2SH-A funding transaction - lack of funds?");
return AcctTradeBot.ResponseResult.BALANCE_ISSUE;
}
try {
bitcoiny.broadcastTransaction(p2shFundingTransaction);
} catch (ForeignBlockchainException e) {
// We couldn't fund P2SH-A at this time
LOGGER.debug("Couldn't broadcast P2SH-A funding transaction?");
return AcctTradeBot.ResponseResult.NETWORK_ISSUE;
}
for(DataCombiner datumToProcess : dataToProcess ) {
// Attempt to send MESSAGE to Bob's Qortal trade address
TradeBotData tradeBotData = datumToProcess.tradeBotData;
byte[] messageData = CrossChainUtils.buildOfferMessage(tradeBotData.getTradeForeignPublicKeyHash(), tradeBotData.getHashOfSecret(), tradeBotData.getLockTimeA());
CrossChainTradeData crossChainTradeData = datumToProcess.crossChainTradeData;
String messageRecipient = crossChainTradeData.qortalCreatorTradeAddress;
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
} else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", datumToProcess.p2shAddress));
}
return AcctTradeBot.ResponseResult.OK;
}
}

View File

@ -1,47 +0,0 @@
package org.qortal.controller.tradebot;
import java.util.Map;
import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
public class TradeStates {
public enum State implements TradeBot.StateNameAndValueSupplier {
BOB_WAITING_FOR_AT_CONFIRM(10, false, false),
BOB_WAITING_FOR_MESSAGE(15, true, true),
BOB_WAITING_FOR_AT_REDEEM(25, true, true),
BOB_DONE(30, false, false),
BOB_REFUNDED(35, false, false),
ALICE_WAITING_FOR_AT_LOCK(85, true, true),
ALICE_DONE(95, false, false),
ALICE_REFUNDING_A(105, true, true),
ALICE_REFUNDED(110, false, false);
private static final Map<Integer, State> map = stream(State.values()).collect(toMap(state -> state.value, state -> state));
public final int value;
public final boolean requiresAtData;
public final boolean requiresTradeData;
State(int value, boolean requiresAtData, boolean requiresTradeData) {
this.value = value;
this.requiresAtData = requiresAtData;
this.requiresTradeData = requiresTradeData;
}
public static State valueOf(int value) {
return map.get(value);
}
@Override
public String getState() {
return this.name();
}
@Override
public int getStateValue() {
return this.value;
}
}
}

Some files were not shown because too many files have changed in this diff Show More