CalDescent
8c3a22aa5c
Improved link replacement criteria.
3 years ago
CalDescent
f3e5933599
Fixed naming error in joinFiles API.
3 years ago
CalDescent
aac4fe37e8
Fixed API response description.
3 years ago
CalDescent
ebfa941a4f
Fixed some file separators.
3 years ago
CalDescent
47c70eea9e
Use system temp directory instead of making a "temp" subfolder when zipping files.
3 years ago
CalDescent
fe7c40cb7c
Reduced log spam.
3 years ago
CalDescent
8973626a4b
Fixed issue with temp directory on Linux.
3 years ago
CalDescent
ace5d999e2
Log a comma separated list of hashes after splitting a file into chunks, so they can easily be requested from another node using the //data/files/frompeer/{peer} API endpoint. Again temporary until the sync happens automatically.
3 years ago
CalDescent
52829a244b
More work on APIs to request files from peers. I won't spend too long making these perfect as they are mostly a temporary measure.
3 years ago
CalDescent
71c247fe56
Added POST /data/file/{hash}/build API, used to join multiple chunks together.
3 years ago
CalDescent
b34066f579
More work on HTML parsing.
...
The style tag parsing ideally needs rewriting using an actual CSS parser, but we can get away with this hacky approach in the short term.
3 years ago
CalDescent
b286c15c51
Optimized website serving, and added code to return the correct content types.
...
This is probably the most efficient way to process the data on the fly, but it's still not very scalable. A better approach would be to pre-process the HTML when building the file structure, and then serve them completely statically (i.e. using a standard webserver rather than via application memory). But it makes sense to keep it this way for development and maybe early beta testing.
3 years ago
CalDescent
ea5e2f5580
Added POST /site/preview API
...
This can be used to preview a site before signing a transaction and announcing it to the network. The response will need reworking to return JSON (along with most of the other new APIs)
3 years ago
CalDescent
b65c7a75fe
Handle relative links when parsing HTML.
3 years ago
CalDescent
39f5dce51c
Moved "directory" data uploads to new POST /site/upload API.
...
Directory uploads don't make much sense outside of website hosting, so it's best to make this API specific to that purpose.
3 years ago
CalDescent
f77ec1faf6
A very crude proof of concept which serves a website from a zipped (and in future, chunked) data blob. This forms the beginnings of the "website hosting" layer on top of the data storage. It needs a significant rework - most importantly so that we aren't serving every asset from memory, and also so that the correct content-type headers are returned, etc.
3 years ago
CalDescent
cd3a1e0159
Increased max file size to 1GiB. Will review this again later.
3 years ago
CalDescent
3f20fadb81
When zipping files, rename the outer folder to "data" instead of using the original folder name. This means that the data can be accessed deterministically without the need to first lookup the folder name.
3 years ago
CalDescent
1c6428dd3b
Added equivalent split and join test but this time using a 5.5MiB source file.
3 years ago
CalDescent
aca620241a
More work on data file split/join, and added a test.
3 years ago
CalDescent
808b36e088
Specify chunk size when splitting.
3 years ago
CalDescent
1613375cc0
Added more validation of files received in GET /data/file/frompeer
3 years ago
CalDescent
787ef957d2
Added support for uploading an entire directory via POST /data/upload/path
...
If a directory is specified instead of a file, the directory is automatically zipped before being split into chunks.
3 years ago
CalDescent
b915d0aed5
Only create the output file directories when we are actually writing a file there. This should prevent empty directories being created when initializing a nonexistent DataFile using a hash.
3 years ago
CalDescent
16dc5b5327
Include the data length in DataFileMessage, which is more similar to the approach used by ArbitraryDataMessage. These two message types are very similar, except arbitrary code currently has a requirement of one piece of data per signature, whereas DataFile code is independent and can support multiple files per transaction. Maybe the two can be combined at some point, but for now I'll keep them separate.
3 years ago
CalDescent
c2d0c63db0
Improved error logging.
3 years ago
CalDescent
f5c9807a48
Use contains() rather than equals() when matching a peer in /data/file/frompeer, so that the port can be optionally left out.
3 years ago
CalDescent
7e9b1d5e16
Rework of DataFile.base58Digest()
...
This fixes an NPE when trying to send a file that doesn't exist. It also removes the caching, which we can add again later if it turns out to be needed.
3 years ago
CalDescent
5070c4eea9
Better handling of data file responses in the /data/file/frompeer API endpoint.
3 years ago
CalDescent
33d9c51b6f
Validate supplied base58 string in /data/file/frompeer API endpoint
3 years ago
CalDescent
d0f9d478c2
Fixed bug which prevented the DATA_FILE message ID from making it through to the reply queue.
3 years ago
CalDescent
f296ec46c8
Removed unused headers.
3 years ago
CalDescent
64d19e480b
Chunk size set to 1MB for now, as it seems that our networking code has problems when transferring 2MB chunks. We can increase this later once that problem has been fixed.
3 years ago
CalDescent
8e35f131d5
Removed debugging log that wasn't intended to be committed.
3 years ago
CalDescent
aafb9d7e4f
Include running total in "Sent X bytes" log entry.
3 years ago
CalDescent
652f30bdbd
Added DATA_FILE and GET_DATA_FILE message types.
3 years ago
CalDescent
f4ba7b2a0c
Added onNetworkGetDataFileMessage() handler
3 years ago
CalDescent
592490d709
Added GET /data/file/frompeer API endpoint
...
This requests a file from the supplied peer address, and stores a copy locally if successful. Still a work in progress.
3 years ago
CalDescent
5ac676d201
Added DataFileMessage and GetDataFileMessage, used for requesting and sending files between peers.
3 years ago
CalDescent
abfe0a925a
More DataFile methods and improvements
3 years ago
CalDescent
fa11f4f45b
Moved DataFileChunk(byte[] fileContent) constructor to the superclass so it can be used by regular data files too.
3 years ago
CalDescent
1e8dbfe4b7
Delete chunk if it fails the hash validation in the constructor.
3 years ago
CalDescent
f82f2bd287
Added DELETE /data/file API endpoint
...
This deletes a file referenced by a user supplied SHA256 digest string (which we will use as the file's "ID" in the Qortal data system). In the future this could be extended to delete all associated chunks, but first we need to build out the data chain so we have a way to look up chunks associated with a file hash.
3 years ago
CalDescent
76742c3869
Removed .dat extension, and use an extra level in the directory structure (data/aB/cD/aBcDeF... etc)
3 years ago
CalDescent
9407e7e418
Data storage location moved to settings ("dataPath")
3 years ago
CalDescent
120552b36e
Added resource file missing from last commit.
3 years ago
CalDescent
9fb58c7ae3
Added the beginnings of an upload API, with some basic data file management.
3 years ago
CalDescent
b917da765c
Removed block 212937
3 years ago
CalDescent
cc59510cd0
Removed all cross-chain code. It's not needed for data nodes.
3 years ago
CalDescent
86aab7023c
Initial settings for data node development. Most to be decided later.
3 years ago