Instead of searching from block 0, we now keep a record of
base trim height in the DB itself.
Also, we no longer trim the latest AT state for non-finished ATs
in case they are in deep sleeping and we need their state for when
they awaken.
Symptoms are:
* db/blockchain.log is pretty much exactly 50MB - the checkpoint-triggering size.
* Loads of threads are stuck waiting for HSQLDB's CountUpDownLatch$Sync.await()
* Synchronizer, or some other thread, possibly orphaning blocks.
The cause seems to be method A, which has a repository session,
calls EventBus.INSTANCE.notify() and one of the event listeners
then obtains their own repository session to do repository 'work'.
In the meantime, the HSQLDB log has reached 50MB, triggering auto-checkpoint.
HSQLDB attempts to CHECKPOINT, but waits for existing transactions
to complete, and also blocks starting new transactions.
Thus, one of the event listeners is blocked when they try to obtain
a new repository session, but HSQLDB never performs CHECKPOINT
because the event notifier (method A) still has an unfinished
transaction - hence deadlock.
Drop created_when column from ATStates as it never changes
and can be fetched from ATs table.
This takes about 50s on a fast machine.
Correspondingly rebuild height-based index on ATStates.
This takes about 3 minutes on a fast machine.
Modify AT-related repository methods and callers.
Aggressively remove 'old' (> 2 weeks) actual AT
state binary data, leaving only the hash in DB
(for syncing purposes). Seems to keep up with
syncing from another node on localhost.
Additional benefit is to speed up syncing if node has trade-bot entries,
as a batch of blocks processed within 30s will have the same HTLC responses
as neither Bitcoin nor Litecoin blocks are that fast. Once synced, the next
Qortal block (~60s) should pick up any new changes. Generally users will be
synced when using trade-bot anyway.
Also moved bitcoiny.getAddressTransactions(p2shAddress)
into BitcoinyHTLC.findHtlcSecret() so only called if secret,
or lack thereof, is not cached.
Added tests to cover caching.
Several cross-chain API calls moved into separate classes,
although most of the URLs remain roughly the same to provide
backwards compatibility.
API /crosschain/at/build moved into /crosschain/BitcoinACCTv1
Converted DELETE /crosschain/tradeoffer to be ACCT-agnostic.
Changes to ACCT interface, etc. to support above.
Changes applied to other crosschain API calls to make them
independent of Bitcoin/Litecoin.
Corrections to fee calculations and usage in BitcoinACCTv1.
Added new LitecoinACCTv1 trade-bot, using LitecoinACCTv1.
Some minor typo corrections, rename of secretHash to hashOfSecret.
Some more Bitcoin-specific fields deprecated, but values duplicated
from newly-named fields for now.
Lower default fee (10sats/byte) for Litecoin spending transactions.
(Not P2SH fees which are 1000sats).
Changed ApiError INSUFFICIENT_BALANCE HTTP status from 422 to 402
as 422 isn't supported by Jetty?
CrossChainTradeSummary.btcAmount deprecated, use: foreignAmount
Modified pom.xml to generated package-info.java files for classes
inside org.qortal.api.model.** subdirectories.
API support for Litecoin wallet balance and sending LTC.
TradeBotCreateRequest rejigged to use blockchain-agnostic
field names, e.g. bitcoinAmount now foreignAmount,
and added foreignBlockchain field.
The massive API CrossChainResource class has been split into:
CrossChainAtResource: for building TRADE/REDEEM/CANCEL messages
(OFFER missing?)
CrossChainBitcoinResource: for Bitcoin wallet balance/spend
CrossChainLitecoinResource: ditto for Litecoin
CrossChainHtlcResource: for Bitcoiny-HTLC actions like:
deriving P2SH address
checking HTLC status
eventually: building refund/redeem transactions
CrossChainResource: for creating/cancelling/listing trade offers.
CrossChainTradeBotResource: for creating/cancelling trade-bot
entries, including responding to trade offers.
---
Other general trading changes:
TradeBot states are now specific to each individual trade-bot,
e.g. BitcoinACCTv1TradeBot or LitecoinACCTv1TradeBot, etc.
TradeBot states now a combination of int & String, instead of
enums due to above.
Extra columns added to DB TradeBotStates to store
blockchain, which ACCT in use, etc.
---
UNTESTED at this point!
Extracted AcctMode from BitcoinACCTv1.Mode as the values are
common to both Bitcoin/Litecoin ACCTs.
Added test apps for deploy, cancel, trade and redeem of LitecoinACCTv1.
Added altcoinj library as Maven dependency.
Added new Litecoin subclass of Bitcoiny,
with mainnet and testnet ElectrumX server lists.
Added litecoinNet settings variable and getter.
Added LitecoinTests.
Most tests work but testFindHtclSecret()
needs a redeemed HTLC on chain (not yet done).
Added litecoinNet to some test settings files
in resources.
Added Litecoin BuildHTLC, Refund test apps.
Added SendLTC app as Electrum-LTC seems a bit flaky?
So far managed to build HTLC P2SH, fund it and then
refund it!
---
As Bitcoin and Litecoin are both subclasses of Bitcoiny,
could unify some test apps with added Bitcoin/Litecoin
switch as first arg?
Bitcoin/Litecoin common aspects extracted in a "Bitcoiny" common class.
So:
Bitcoin (was BTC) extends Bitcoiny
Litecoin (future code) will also extend Bitcoiny
ElectrumX is now a BitcoinyBlockchainProvider
to allow easier future replacement and also tidier integration.
BTCP2SH is now BitcoinyHTLC as they are generic hash time-locked contracts,
probably Bitcoin/Litecoin agnostic.
BTCACCT is now BitcoinACCTv1, allowing for v2+ and also LitecoinACCTv1, etc.
BitcoinTransaction is now BitcoinyTransaction
as they are pretty much the same in Litecoin.
BitcoinException is now a more generic ForeignBlockchainException.
---
Bitcoiny subclasses instantiate a new BitcoinyBlockchainProvider
when creating their singleton instance. They pass relevant network
details to the BBP, like server lists, genesis block hash, etc.
Bitcoiny.WalletAwareUTXOProvider now only has the one key search mode
that is equivalent to the old REQUEST_MORE_IF_ANY_SPENT.
Tests tidied up.
---
Still to do:
Modifying TradeBot to handle multiple types of ACCTs,
like BitcoinACCTv2, LitecoinACCTv1...
Modifying API to support multiple types of ACCTs.
Actually add Litecoin support.
Build new ACCT without needing P2SH-B if possible.
There's still an existing issue where log entries like this appear:
Unable to trim old online accounts signatures in repository
which is actually caused by:
integrity constraint violation: unique constraint or index violation; SYS_PK_10092 table: BLOCKS
which seems to be a bug in the version of HSQLDB we use.
(Tested using synced-from-scratch DB).
It's not clear what the actual problem is at this point.
It might be possible to switch to v2.5.1 if our recent HSQLDB-related
commits have fixed/worked-around the OOM issues.
Move the inner method from BlockChain to Controller.
Remove blockchain lock as it's not needed because it's not an
HSQLDB "serialization failure" but constraint violation.
Trimming old online accounts signatures limited to batches of 1440
rows to reduce CPU and memory load.
Added separate method to determine status of P2SH transactions,
returning UNFUNDED, FUNDING_IN_PROGRESS, REDEEMED, etc.
Added code to trade-bot to increase robustness. Lots more
changes including unified state change/logging, checking
for existing MESSAGEs, etc.
Added missing websocket methods to silence log noise.
Trade-bot now called per block during synchronization,
instead of per batch, to pick up edge cases where some
potential trade-bot transitions were missed, resulting
in failed trades.
Corresponding changes in Controller, such as notifying
event bus of new block in same thread (thus blocking)
instead of using executor.
Added slightly more robust common block determination
to Synchronizer.
Refactored code in BTC class to use new BitcoinException
rather than simply returning null, with added sub-classes
allowing differentiation between network issues or fund
issues.
Changed BTC.buildSpend to try harder to find UXTOs to
address false "insufficient funds" issues.
Repository change to add index on MessageTransactions
for quicker look-up of trade-related messages.
Reduced reliance on bitcoinj library in BTCP2SH.
Reworked ElectrumX to better detect errors rather than
continuously try more servers to no avail.
Also added genesis block check in case of servers on
different Bitcoin networks.
Now tries to extract upstream bitcoind error codes
and pass those up to caller via exceptions.
Updated list of testnet servers.
MemoryPoW now detects thread interrupt and exits fast.
Moved some non-generic transaction-related repository
methods to their own subclass. For example:
moved TransactionRepository.getMessagesByRecipient
to MessageRepository.getMessagesByParticipants
Updated and added more tests.
Was "run.sh" but renamed to "start.sh" to better complement "stop.sh".
"run.sh" is now a symbolic link to "start.sh"
Reworked Java version check to remove dependency on "bc" tool which
seems not to be installed on some Ubuntu distributions?
Removed -XX:NativeMemoryTracking flag from JVM args.
Fixed incorrect comment regarding java.net.preferIpV4Stack.
Fixed typo in comment.
catbref