Skip to content

Commit c137b1b

Browse files
astaphobiailtogaandy-shi88sg2bczsukrawidhyawan
authored
Snapshot, Multisig, Escrow Approval, Fee Model, Bug Fixes (#737)
* Code Refactoring (#587) * Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths * Get escrow transactions with filter fields and pagination (#590) * get escrow transactions with filter fields and pagination * get escrow transactions functional test * #588 decouple published receipt and participation score from block main service (#591) * 592 fee model (#595) * #592 add fee model to action type * #592 add fee model to action type * #592 ignore package-interface naming lint rule * #592 update pb file of transaction : add model + service for getTxMinimumFee api * #592 add handler for calculate minimum fee api * #592 fix GetTransactionMinimumFee request * #592 fix wrong type * #592 add api client for testing get minimum fee * 578 snapshot file creation (#593) * Logic for snapshot file creation * Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain) * Add ugorji/go/codec package to add CBOR serialization support * Added Logic for snapshot file processing (create/download) * Add new interface and service to abstract snapshot file download * Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks * Added business logic for snapshots * Add new service FileService to abstract (and make testable) methods relative to file manipulation * Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods) * Move GetFileNameFromHash and GetHashFromFileName into fileservice * Add query and service handler to retrieve account balances for snapshots * Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP) * Add initial logic to create a snapshot file containing all (only last state) account balances * TODO: increase code coverage (few methods are tested so far) * Bug fix * Add EscrowTransactionQuery to GetDerivedQueries * Added logic for generating snapshot files * Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService * Add test case for NewSnapshotFile and new queries * Finalised logic for generating snapshot files (first iteration only has single file per snapshot) * Add all relevant queries to get snapshot's payload * Add method to parse and import snapshot into db * Add config parameter for snapshots folder * Add unit and integration tests * Added logic for download snapshots * Add method to parse the spineBlockManifest and download file chunk hashes * TODO: download file is still mocked * Uncomment logic previously commented for testing purposes * Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments * Add query interface for snapshot-specific queries (SnapshotQuery) * Add SelectDataForSnapshot interface signature to new interface * Add GetSnapshotQuery method to get all snapshot queries implementations (in a map) * Refactor relative query methods in every relevant repository * Refactor some constant to use time.Duration instead of int type * Fix unit tests * Add DESC to order by clauses in snapshot queries * Refactor GetChainTypeCount to GetChainTypes to make chain types countable * Remove temporary test function * Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication) * Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication) * 596 multisig skeleton (#597) * #596 add multisignature transaction body * #596 add constants for MultiSignature body fields * #596 add multisignature tx type skeleton * #596 add multisig tx type enum * #596 add command tools for generate multisignature tx and parsing any tx * #596 basic multisignature parsing functionality * #596 update schema commit * #596 add signature info * #596 update signature structure to use map * #596 adjust cmd tools to new multisig byte structure * #596 update multisig transaction bytes * #596 update schema * 599 snapshot chunk strategy (#605) * Added proto message for snapshots * Add SnapshotPayload message representing internal snapshot data structure * Added logic for snapshot chunk 'basic' strategy * Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot * Add SnapshotPayload model to schema, since we are using it in two different packages * Refactor snapshotmainblockservice to use new snapshot chunk strategy * Refactor unit and integration tests * Completing logic for snapshot basic chunk strategy * Finalize BuildSnapshotFromChunks and its test * Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service * Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile * 600 multisig queries (#607) * #600 add new managed table for multisig * #600 update multisig managed table model * #600 add managed table query for multisig Co-authored-by: Ariasa Ketut <[email protected]> * 602 generate multisig address (#609) * method multisig account address assembler * command line tool multisig generator command * set 1.20.0 of golangci-lint version to fixed circleci fail * no need to validate current block height (#610) * inhibit spine blocks smithing process during main blocks first download (#608) * Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started) * Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService * Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service * Add new blocker type P2PNetworkConnectionErr * Increase timers for checking block download process * Refactor blockStatusService and implementations to blockTypeStatusService * Refactor relative code in other services * Remove cast to int snapshotPayloadHeight in snapshotMainBlockService * Fix bug in snapshotPayloadHeight computation * Refactor constructor method for BlockchainSyncService * Fixed transaction generator affected by fee (#611) * no need to validate current block height * fixed transcation generator afftected by fee * Code Refactoring and Bug Fixing (#613) * Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded) * Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status * Remove unused getter from chainType * Add new constants for better code readability * Fix tests * 601 multisig validation (#615) * #601 add --post and --post-host to enable post transaction bytes on generated * #601 add multisig validation * #601 fix lint * #601 wrong description in cmd * Optimize GRPC duration (#616) * no need to validate current block height * set index into some tables for speed up latency get a lot of records * remove unused a line of codes * GRPC endpoint get single escrow transaction by ID (#620) * no need to validate current block height * functionality test get escrow by id * Download snapshot chunks (#614) * Change fatalf to errorf/return in goroutine in snapshotService.go * Add missing return statement for error in snapshotService.go * Added logic for downloading files from a random peer and Code Refactoring * Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy * Add rpc call (client request and server response) to download a file from a random peer * Add logic to download/parse/verify a file and a snapshot * Added logic and file download metrics * Add logic to download a file, server side * Add metrics to count succeeded/failed snapshot downloads * Update schema: add Failed files for file download * Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node * Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones * Rename megablockQuery to spineBlockManifestQuery * Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex * Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data * Fix smith process lock when downloading spine and main blocks * Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing * Add retry download failed file chunks from different peers * WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string * Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them * Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases) * Optimize p2p.DownloadFilesFromPeer while writing the test * Fix code after code review * fixed bug in caclulating snapshot's height * Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count) * Refactor code that used sync.Map in favor of new map struct * Add util function to calculate a quick (non secure) random number * Fix uncaught nil in p2pServerHandler * Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks) * Add a couple of TODOs for spine block manifest validation * Fix bug in unitialized model.MapIntBool struct * Add missing test for GenerateSnapshot method * Uncommented code for testing * Uncomment code * Fix one unit test after fixing the snapshot height (previous commit) Co-authored-by: stefano galassi <[email protected]> * 594 add bitcoin signature (#618) * add bitcoin sig, btcd & btcutil package * restructure Ed25519 signature type function * add cmd to generate bitcoin account * add signature-type flag in cmd tx generator * cmd enhace switch case generate account * return error when sign and verify payload * cmd generate tx with sender signature type * adjust general parse transaction bytes * adjust unit test * udate cmd readme * cmd, panic when invalid signature type * remove unused function * add generate account in signature * add signature unit test * add misisng import package * update schema * change parameter type of GenerateAccountFromSeed Co-authored-by: Ariasa Ketut <[email protected]> * 629 - Use gomod consistently (#630) * no need to validate current block height * update circleci config use go modules instead of dep and update go version to 1.14 * go mod and more cleaning shadow declaration variable * Revert "GRPC endpoint get single escrow transaction by ID (#620)" This reverts commit 7824924. * Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)"" This reverts commit bedbf87. * revert unused commit * remove Gopkg.* files * remove no needed comment * make sure go modules cache correctly * change circleci working directory, update readme * update path of cache * fixed go modules caches * 603 multisig apply (#623) * #603 initialize multisig service layer * #603 implement apply confirmed * #603 add sender address in pending transaction * #603 update multisignature.PendingTransaction schema * #603 delete multisig tx util * #603 add sign command to support multisig development * #603 add --sender-address flag for generating unsigned bytes * #604 add sender address to unsigned tx * #603 return raw error from execute transaction to enable sqlite error casting * #603 complete multisig implementation * #603 fix tests * #603 add latest flag to pending transaction * #603 update pending_transaction status on executed * #603 add latest flag to pending transaction * #603 add filter to pending transaction query * #603 add latest flag to multisig-info and signatureInfo * #603 add multisig-child flag to tx table * #603 add multisig-child flag to tx table * #603 add multisig-child flag to tx model * #603 add latest field to all pending tables * #603 adjust query to handle latest flag * #603 insert multisig-child transaction to transaction table * #603 dfix test * #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig) * #603 fix example cmd * #603 update schema * #603 remove binary * #603 convert address-Signature to format stringTostring * #603 fix query inconsistency * #603 handle unsigned transaction execution status * #603 revert unnecessary change * #603 handle submit multisig info execute multiple tx * #603 fix query inconsistency * #603 add multisig signature type * #603 add constant for multisig signature validation type; * #603 remove debug log * CI delivery image version (#632) * fixed invalid config * change image version to stretch * remove configuration for test purposes * Ci delivery image version (#633) * fixed typo * 604 api rate limit (#625) * add rate limiting feature for API * fix unclosed rows * change MaxAPIRequestPerSecond from int32 to uint32 * add download process metrics tracker * add go sum * 621 - Respecting escrow expiration on Push Block (#626) * no need to validate current block height * fixing bug, respecting expiring escrow process on PushBlock * fixed escrow rollback query and respecting escrow stage before push block process * fixed escrow unit test result * has been fixed forgot to close rows * allow expiring escrow method to control db transaction * Spine block manifest and spine public keys validation when downloading blocks from peers (#631) * Bug fixing * Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers * Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash) * Fix tests * Code Refactoring * Refactored method's name from GetPayloadHash to GetPayloadHashAndLength * Added logic to validate block's payload on block push * Add ValidatePayloadHash logic to blockMainService and blockSpineService * Fix tests * Add logic to validate spine block manifest against its relative spine block of reference * Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp) * Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network * Add unit tests for new methods * Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation # Conflicts: # core/service/blockMainService.go * Code Refactoring * Update genesis block ID after changes to Generate block (added published receipts to payload length) * Add unit test for validateblockhash * Code Refactoring * Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods * Code Refactoring * Removed unnecessary comment * Bug fixing * Remove fmt.println probably left from some internal test * Bug fixing * Update comment in query * Code Optimization * Optimize if statement by reversing conditions (courtesy of @ali) Co-authored-by: stefano galassi <[email protected]> * Bug Fixing (#636) * Fixed bug in download snapshot process causing segfault Co-authored-by: stefano galassi <[email protected]> * Bug Fixing (#639) * Fix bug in pendingSignatureQuery rollback function * Remove unused function from transactionQuery Co-authored-by: stefano galassi <[email protected]> * Escrow query test fail fixed (#640) * no need to validate current block height * escrow query test fail fixed * Bug Fixing (#642) * Fix bug in unchecked map key leading to panic in block smith process * Escrow query test fail fixed (#647) * no need to validate current block height * escrow query test fail fixed * fixed escrow map params need to sorting before compare * 634 multisig api (#645) * #634 init multisig api * #634 update multisig request model * #634 update get pending tx detail response * #634 update get pending tx detail response * #634 initiate multisig client API * #634 fix bug: wrong empty signature checks * #634 remove built binary from repository * #634 add multisig api tests * #634 revert unnecessary changes * #634 update go mod * #634 initiate health check service and model * #646 add health check endpoint * #634 update schema * #634 use getter instead of accessing field directly * #634 add stricter param check * #634 fix pagination * #634 remove duplicate code Co-authored-by: Ariasa Ketut <[email protected]> * 634 multisig api (#653) * #634 add default field to order field param Co-authored-by: Ariasa Ketut <[email protected]> * #651 fix multisig_address signature validation (#652) * #651 fix multisig_address signature validation * #651 fix test Co-authored-by: Ariasa Ketut <[email protected]> * 656 incorrect account ledger balance changes (#657) * no need to validate current block height * account ledger incorrect balance change information fixed * 654 account ledger multisig (#658) * #654 add multisig transaction event to account ledger * #654 add ledger record in multisig transaction * #654 update schema commit Co-authored-by: Gede Sukra Widhyawan <[email protected]> * 660 api get multisig info (#661) * #660 add new api endpoint proto for getting multisignature info * #660 implement get multisig info api * #660 rename getPendingTransactions request * #660 rename getPendingTransactions request * #660 update schema commit * 650 - CMD HD wallet generate account (#668) * no need to validate current block height * generate account slip10 * allow fetch from private repo * fixed invalid format of ci * fixed set env of machine * add additional ssh key * fixed bash condition when check gomod exists * allowed github connection via token instead of ssh * remove unused code * more describe generate account command * fix uninitialized metrics (#662) fix uninitialized metrics (#662) * bitcoin sig: add public key into signature (#670) * bitcoin sig: add public key into signature * handle multiple rows in a function (#675) handle multiple rows in a function (#675) * SelectReceipt Limit (#676) SelectReceipt Limit (#676) * Circleci GitHub token (#677) * token as env variable * 628 apply snapshot (#667) * added alpha nodes genesis.go * Rollback mempool - Restoring mempool transactions backup (#419) * initiate restoreMempoolsBackup func * finished mempool backup process and add loggger into processfork struct * add logger into blockPopper stuff * 420 get scramble nodes enhance (#423) * #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now * #420 move some scrambling function to util to reduce fetching function call frequency * #420 add block_height to merkle tree table * #420 fix query to fetch receipt, it had wrong block height scope on the query * #420 handle error * #420 add receipt validation * #420 build scrambled node at height x * #420 adjust constructor update * #420 move receipt service instantiation to after node registration service * #420 move scramble node model outside, it is used by other service as well : receipt service * #420 validate receipt before storing to database * #420 use bytes.equal for better comparison performance * #420 rename memoized scramble nodes * #420 simplify loop * 424 fix add participation score query (#425) * fixed addParticipationScore query refactored unit test refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) ) * removed unused InsertParticipationScore query * refactored addparticipationscore query to use a single upsert statement * Remove some process (#429) * disable some process to adjust with alpha * update unit test * reverted broken query and kept changes from newer pr * 432 revert participation score query (#437) * 424 fix add participation score query (#425) * fixed addParticipationScore query refactored unit test refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) ) * removed unused InsertParticipationScore query * refactored addparticipationscore query to use a single upsert statement * Remove some process (#429) * disable some process to adjust with alpha * update unit test * reverted query * reverted broken participation score query * change a few configurations special for alpha (#434) * change a few configurations special for alpha * updated config.toml * added (peer) port to genesis nodes addresses * fixed test * updated production values for some constants * updated log levels for testing alpha nodes * reset default log levels * increased PriorityStrategyBuildScrambleNodesGap to 40 * fixed unwanted diffrences between develop and alpha branches * fixed unit tests for alpha * fixed unit test for mempoolCoreService * Fixed unit test * Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks * switch log level for debugging purpose * debug: process fork warn to errorf log * revert: revert log levels * keep track of p2p go routine metrics (#569) * fix shadowed error message * Resolved merge conflicts * Code refactoring * Updated mainchain genesis id * Code Refactoring * Update main block genesis ID * Updated constant * Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails * Update golang.org/x/crypto/ed25519 to crypto/ed25519 * Reverted crypto/ed25519 to golang.org/x/crypto/ed25519 * Fixed bug in download snapshot process causing segfault * Adding business logic * Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo * Add logic to snapshot service to include multisig data in snapshots * Added business logic * Add TrimDataBeforeSnapshot to all SnapshotQueries * Added business logic * Add TrimDataBeforeSnapshot to all SnapshotQueries * Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported) * Add logic to main.go to import snapshot after downloading it * Refactor IsSnapshotHeight with simplified logic * Refactor code to be aligned with develop branch instead of alpha * Refactor SelectDataForSnapshot queries to use rollback-like logic * Refactor SelectDataForSnapshot queries to use rollback-like logic * Write unit tests for new queries * Fix wrong table nale in new block query * Fix wrong table nale in new block query * Fix do not allow to delete genesis block when trimming block table * Revert configX.toml log level to "fatal", "error", "panic" * #628 add checks if all blocksmith time has expired * #628 move push block to anonymous function * Add rollback to snapshot height to set latest = 1 to all latest entitites * Fixed SelectDataForSnapshot query in accountDataset repo * Fix missing for loop break conditions in ScanBlockPool * Add function to log running binary file checksum at node startup Co-authored-by: stefano galassi <[email protected]> Co-authored-by: Andy Shi <[email protected]> Co-authored-by: Ahmad Ali Abdilah <[email protected]> Co-authored-by: stefano galassi <[email protected]> * 672 makefile binary built (#678) * #672 ignore release folder * #672 initialize makefile * #672 add github token example file * #672 move xgo to zoobc repository * #672 add makefile usage * #672 add cc for cmd * #672 add go-fmt and separate golangcilint version * #672 move test to makefile command * #672 fix gopath * #672 update readme for github.token usage * #672 print information for step ran Co-authored-by: Ahmad Ali Abdilah <[email protected]> * 665-Enhance CMD Database Init (#680) * cmd: put init db only where is needed * cmd: add commad to generate proof of ownership * cmd: add readme rollback * cmd: fixing remove node reference func * handle nil value (#682) * throw error on over limit get blocks (#684) * throw error on over limit get blocks * move limit validation to handler layer * remove pending tx status change, it has been updated in apply confirmed (#685) * 686: Cmd, enhance generate account (#687) * cmd: enhance generate account * add get public key using slip10 * 655 escrow receipt approval (#688) * no need to validate current block height * account dataset escrow approval and integration * fullfill send money validation test cases * to be gentle no need to check the account dataset property * create specific method of query for account dataset * getHostInfo unit testing (#689) * getHostInfo unit testing Co-authored-by: Nawi Kartini <[email protected]> Co-authored-by: Ariasa Ketut <[email protected]> Co-authored-by: Ahmad Ali Abdilah <[email protected]> * adding database status metrics (#693) adding database status metrics (#693) * Bug Fixing snapshot selection queries and enhancing rollback queries (#696) * Bug Fixing * Fix snapshot selection queries * Enhance rollback queries * Enhance db tables by adding composite primary keys where possible * Fix bug in rollback and snapshot selection queries Co-authored-by: stefano galassi <[email protected]> Co-authored-by: Andy Shi <[email protected]> * 681 disable smithing non registered node (#694) * #681 add zero participation score blocker * #681 inject blockchainStatusService to noderegistration service * #681 add flag to say if a node is blocksmith or not * #681 update main.go with new struct constructor * #681 only update the blocksmith status when detect zero participation score * #681 compare with bytes.equal() * unit testing for GetBlockQueue (#698) unit testing for GetBlockQueue (#698) * Bug Fixing (#706) * Fix breaking bug in rollback query * Refactor spinePublicKey rollback query to comply to new code logic * 691 grpc account dataset (#699) * no need to validate current block height * rename accountDataset to plural name * account dataset grpc service list and single record * grpc client example * move params checkup to handler package * remove unnecessary test cases * technical error message passed * 690 empty block smith (#703) * #690 add empty flag to enable empty block generate * #690 add limit for skipped block smith * #690 set the skipped limit back to 15 * #690 remove unimplemented test * #690 fix return arguments * #690 remove unnecessary variable initialization * #690 make use of default int64 value * #690 smaller skipped blocksmith constant so we can see the effect on dev net * adding function for HD wallet testing (#683) * adding function for HD wallet testing * P2P: Add unit test on server service (#707) * p2p: add unit test p2p server service * enhance Err format * change GetPeerInfoResponse field name * p2p: update unit test * update schema Co-authored-by: Andy Shi <[email protected]> * added skippedblocksmith to snapshot (generate/apply) (#710) * Add SkippedBlocksmith to snapshot proto model * Added Logic for snapshots * Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository * Added Logic for snapshots * Add logic to include skippedblocksmith records to snapshots * Add logic to import skippedblocksmith records from snapshots into db (apply snapshot) * Fix unit tests * Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot Co-authored-by: stefano galassi <[email protected]> * Adding runtime cpu profiling for core node (#712) * Adding Application Logic * Add configuration parameter for profiling * Add runtime flag to enable profiling * Add logic to run cpu profiling * Update configuration parameter for profiling * Add logic to enable memory profiling * Update documentation (readme.md): see 'run' paragraph * add more info in log (#713) * fixed inconsistance block height on escrowed transaction (#717) * Code Refactoring (#716) * Use net/http/pprof package that serves the full profiling api via http * Enable profiling api only when running app with -cpu-profile flag (see updated readme) * Refactor the way monitoring server is instantiated: not using default http hanlder anymore * Update readme.md Co-authored-by: stefano galassi <[email protected]> * Code Refactoring (#719) * Refactor block query to select a range of blocsk instad of a single one for snapshots * Refactor service logic to add more blocks (720) to snapshots * Cleaning redundant function (#714) * adding function for HD wallet testing * cleaning some unused variables * remove unused const and redundant function on generator * Bug Fixing (#724) * Fix pprof http server listening only on localhost Co-authored-by: stefano galassi <[email protected]> * Code Refactoring (#728) * Optimize function GenerateTxRemoveNode Co-authored-by: stefano galassi <[email protected]> * 722 blockchain orchestrator (#726) * add BlockchainOrchestrator to control the chains * 709 Smithing All Blocksmith Skipped (#725) * #709 add smithing pending blocker * #709 remove smith time from blcoksmith model * #709 update smithing process: recycle blocksmith when all blocksmith time are passed * #709 separate smith time constants between spinechain and mainchain * #709 inject spinechain blocksmith service * #709 update metric smithtime to smith index * #709 cover new smithing process unit tests * #709 fix comment on blocksmith strategy & update fake smith function signature * add index on block table and update sql connnection number (#729) add index on block table and update sql connnection number (#729) * scale down the common block jump (#731) scale down the common block jump (#731) * 721 - Remove Account Dataset Expiration (#732) * accountDatasetsQuery should a plural name * update affected services and more about account_dataset stuff * fixing unit test of NewSnapshotFile integration * escrow validation timeout limit exceeded * Add node registry partial history to snapshots (#734) * Code Refactoring * Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process * Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter * Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve: * latest state from block 0 to 'fromHeight' * full history fromHeight to toHeight * merge and remove duplicate ids in first selection * Remove unused constant LinkedReceiptBlocksLimit Co-authored-by: stefano galassi <[email protected]> * making the download process less restrictive to failures in catching up with peers (#735) * making the download process less restrictive to failures in catching up with peers * fix the way of tracking errors * fix the way of keeping track the error * TestBlockIncompleteQueueService_AddTransaction (#720) * TestBlockIncompleteQueueService_AddTransaction * add succes case * fixed test case spine generate genesis block Co-authored-by: iltoga <[email protected]> Co-authored-by: Andy Shi <[email protected]> Co-authored-by: stefano galassi <[email protected]> Co-authored-by: Gede Sukra Widhyawan <[email protected]> Co-authored-by: Ahmad Ali Abdilah <[email protected]> Co-authored-by: Nawi Kartini <[email protected]> Co-authored-by: stefano galassi <[email protected]> Co-authored-by: Kevin Hadi <[email protected]>
1 parent d16cb69 commit c137b1b

File tree

178 files changed

+10923
-3830
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

178 files changed

+10923
-3830
lines changed

.circleci/config.yml

Lines changed: 22 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,28 @@
11
version: 2
2+
defaults: &defaults
3+
working_directory: ~/zoobc
4+
docker:
5+
- image: circleci/golang:1.14.0-stretch
26
jobs:
37
prepare:
4-
working_directory: ~/zoobc
5-
docker:
6-
- image: circleci/golang:1.14.0-stretch
8+
<<: *defaults
79
steps:
810
- run:
9-
name: SSH
10-
command: git config --global url.ssh://[email protected]/zoobc.insteadOf https://github.com/zoobc
11+
name: GIT CONF
12+
command: |
13+
git config --global url."https://github:[email protected]".insteadOf "https://github.com"
1114
- checkout
15+
- add_ssh_keys
16+
- run:
17+
name: ENV SET
18+
command: |
19+
echo 'export GOPRIVATE=github.com/zoobc/*' >> $BASH_ENV
1220
- restore_cache:
1321
key: gopkg-{{ .Branch }}-{{ checksum "go.sum" }}
1422
- run:
1523
name: GOMOD
1624
command: |
17-
if [ ! -d ls | grep go.mod ]; then
25+
if ! [ -f "go.mod" ]; then
1826
go mod init
1927
fi
2028
go mod download
@@ -23,31 +31,17 @@ jobs:
2331
paths:
2432
- /go/pkg/mod
2533
test:
26-
working_directory: ~/zoobc
27-
docker:
28-
- image: circleci/golang:1.14.0-stretch
29-
environment:
30-
- GOLANGCI_LINT: 1.20.0
34+
<<: *defaults
3135
steps:
3236
- checkout
3337
- restore_cache:
3438
key: gopkg-{{ .Branch }}-{{ checksum "go.sum" }}
35-
- run:
36-
name: ↓ GOLANGCI-LINT
37-
command: |
38-
curl -sfL https://github.com/raw/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v${GOLANGCI_LINT}
3939
- run:
4040
name: ↓ GOLINT
4141
command: go get -u golang.org/x/lint/golint
4242
- run:
43-
name: TEST
44-
command: go test `go list ./... | egrep -v 'common/model|common/service'` --short
45-
- run:
46-
name: GO FMT
47-
command: go fmt `go list ./... | egrep -v 'common/model|common/service|vendor'`
48-
- run:
49-
name: GOLANGCI-LINT RUN
50-
command: golangci-lint run
43+
name: GO Test
44+
command: make test
5145
- run:
5246
name: GOLINT RUN
5347
command: golint `go list ./... | egrep -v 'vendor|common/model|common/service'`
@@ -56,9 +50,7 @@ jobs:
5650
paths:
5751
- /go/pkg/mod
5852
build:
59-
working_directory: ~/zoobc
60-
docker:
61-
- image: circleci/golang:1.14.0-stretch
53+
<<: *defaults
6254
steps:
6355
- checkout
6456
- restore_cache:
@@ -77,15 +69,13 @@ jobs:
7769
name: BUILD
7870
command: |
7971
go build -o ./dist/zoobc
80-
# rsync -va --exclude='*.db' ./resource ./dist
72+
# rsync -va --exclude='*.db' ./resource ./dist
8173
- save_cache:
8274
key: build-cache-{{ .Branch }}-{{ .Environment.CIRCLE_SHA1 }}
8375
paths:
8476
- ./dist
8577
deploy-develop:
86-
working_directory: ~/zoobc
87-
docker:
88-
- image: circleci/golang:1.14.0-stretch
78+
<<: *defaults
8979
steps:
9080
- checkout
9181
- add_ssh_keys
@@ -115,9 +105,7 @@ jobs:
115105
command: |
116106
for host in $ln1 $ln2 $ln3; do ssh root@$host 'sudo systemctl start zoobc.service'; done
117107
deploy-staging:
118-
working_directory: ~/zoobc
119-
docker:
120-
- image: circleci/golang:1.14.0-stretch
108+
<<: *defaults
121109
steps:
122110
- checkout
123111
- add_ssh_keys
@@ -169,4 +157,4 @@ workflows:
169157
- build
170158
filters:
171159
branches:
172-
only: staging
160+
only: staging

.editorconfig

Lines changed: 0 additions & 12 deletions
This file was deleted.

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@ coverage.*
1818
resource/zoobc_*
1919
resource/snapshots*
2020
resource_cluster/zoobc_*
21+
resource/*.prof
2122
cmd/*.new
2223
accounts.txt
24+
.editorconfig
2325
.manual
26+
release/
27+
github.token

.golangci.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ run:
7676

7777
issues:
7878
exclude-rules:
79+
- text: "should be"
80+
linters:
81+
- stylecheck
7982
- text: "rows.Err must be checked"
8083
linters:
8184
- rowserrcheck

Makefile

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
BIN_DIR := $(GOPATH)/bin
2+
GOLANGCILINT := $(BIN_DIR)/golangci-lint
3+
GOLANGCILINT_VERSION := v1.20.0
4+
XGO := $(BIN_DIR)/xgo
5+
VERSION ?= latest
6+
BINARY_CORE := zoobc
7+
BINARY_CLI := zoomd
8+
GITHUB_TOKEN ?= $(shell cat github.token)
9+
10+
.PHONY: test
11+
test: go-fmt golangci-lint
12+
$(info running unit tests...)
13+
go test `go list ./... | egrep -v 'common/model|common/service'` --short
14+
15+
$(GOLANGCILINT):
16+
$(info fetching golangci-lint...)
17+
curl -sSfL https://github.com/raw/golangci/golangci-lint/master/install.sh | sh -s -- -b $(GOPATH)/bin $(GOLANGCILINT_VERSION)
18+
19+
$(XGO):
20+
$(info fetching zoobc/xgo...)
21+
go get github.com/zoobc/xgo
22+
23+
.PHONY: golangci-lint
24+
golangci-lint: $(GOLANGCILINT)
25+
$(info running linter...)
26+
golangci-lint run
27+
28+
.PHONY: go-fmt
29+
go-fmt:
30+
$(info running go-fmt...)
31+
go fmt `go list ./... | egrep -v 'common/model|common/service|vendor'`
32+
33+
.PHONY: build
34+
build:
35+
$(info build core with host os as target...)
36+
mkdir -p release
37+
go build -o release/$(BINARY_CORE)-$(VERSION)
38+
39+
.PHONY: core-linux
40+
core-linux: $(XGO)
41+
$(info build core with linux as target...)
42+
mkdir -p release
43+
xgo --targets=linux/amd64 -out=release/$(BINARY_CORE)-$(VERSION) --go-private=github.com/zoobc/* --github-token=$(GITHUB_TOKEN) ./
44+
45+
.PHONY: core-windows
46+
core-windows: $(XGO)
47+
$(info build core with windows as target...)
48+
mkdir -p release
49+
xgo --targets=windows/* -out=release/$(BINARY_CORE)-$(VERSION) --go-private=github.com/zoobc/* --github-token=$(GITHUB_TOKEN) ./
50+
51+
.PHONY: core-darwin
52+
core-darwin: $(XGO)
53+
$(info build core with darwin/macos as target...)
54+
mkdir -p release
55+
xgo --targets=darwin/* -out=release/$(BINARY_CORE)-$(VERSION) --go-private=github.com/zoobc/* --github-token=$(GITHUB_TOKEN) ./
56+
57+
.PHONY: cmd-darwin
58+
cmd-darwin: $(XGO)
59+
$(info build cmd with darwin/macos as target...)
60+
mkdir -p cmd/release
61+
xgo --targets=darwin/* -out=cmd/release/$(BINARY_CORE)-$(VERSION) --go-private=github.com/zoobc/* --github-token=$(GITHUB_TOKEN) ./cmd/
62+
63+
.PHONY: cmd-linux
64+
cmd-linux: $(XGO)
65+
$(info build cmd with linux as target...)
66+
mkdir -p release
67+
xgo --targets=linux/amd64 -out=cmd/release/$(BINARY_CORE)-$(VERSION) --go-private=github.com/zoobc/* --github-token=$(GITHUB_TOKEN) ./
68+
69+
.PHONY: cmd-windows
70+
cmd-windows: $(XGO)
71+
$(info build cmd with windows as target...)
72+
mkdir -p release
73+
xgo --targets=windows/* -out=cmd/release/$(BINARY_CORE)-$(VERSION) --go-private=github.com/zoobc/* --github-token=$(GITHUB_TOKEN) ./
74+
75+
.PHONY: release-core
76+
release-core: core-linux
77+
78+
.PHONY: release-cmd
79+
release-cmd: cmd-linux

api/api.go

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ import (
66
"net"
77
"net/http"
88

9-
grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
9+
grpcMiddleware "github.com/grpc-ecosystem/go-grpc-middleware"
1010
"github.com/grpc-ecosystem/grpc-gateway/runtime"
1111
log "github.com/sirupsen/logrus"
1212
"github.com/zoobc/zoobc-core/api/handler"
@@ -56,7 +56,7 @@ func startGrpcServer(
5656
}
5757
grpcServer := grpc.NewServer(
5858
grpc.Creds(creds),
59-
grpc.UnaryInterceptor(grpc_middleware.ChainUnaryServer(
59+
grpc.UnaryInterceptor(grpcMiddleware.ChainUnaryServer(
6060
interceptor.NewServerRateLimiterInterceptor(constant.MaxAPIRequestPerSecond),
6161
interceptor.NewServerInterceptor(
6262
logger,
@@ -167,6 +167,14 @@ func startGrpcServer(
167167
)})
168168
// Set GRPC handler for health check
169169
rpcService.RegisterHealthCheckServiceServer(grpcServer, &handler.HealthCheckHandler{})
170+
171+
// Set GRPC handler for account dataset
172+
rpcService.RegisterAccountDatasetServiceServer(grpcServer, &handler.AccountDatasetHandler{
173+
Service: service.NewAccountDatasetService(
174+
query.NewAccountDatasetsQuery(),
175+
queryExecutor,
176+
),
177+
})
170178
// run grpc-gateway handler
171179
go func() {
172180
if err := grpcServer.Serve(serv); err != nil {
@@ -243,5 +251,6 @@ func runProxy(apiPort, rpcPort int) error {
243251
_ = rpcService.RegisterEscrowTransactionServiceHandlerFromEndpoint(ctx, mux, fmt.Sprintf("localhost:%d", rpcPort), opts)
244252
_ = rpcService.RegisterMultisigServiceHandlerFromEndpoint(ctx, mux, fmt.Sprintf("localhost:%d", rpcPort), opts)
245253
_ = rpcService.RegisterHealthCheckServiceHandlerFromEndpoint(ctx, mux, fmt.Sprintf("localhost:%d", rpcPort), opts)
254+
_ = rpcService.RegisterAccountDatasetServiceHandlerFromEndpoint(ctx, mux, fmt.Sprintf("localhost:%d", rpcPort), opts)
246255
return http.ListenAndServe(fmt.Sprintf(":%d", apiPort), mux)
247256
}

api/client/GenerateNodeKey/client.go

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ import (
77
"fmt"
88
"time"
99

10-
"github.com/sirupsen/logrus"
1110
log "github.com/sirupsen/logrus"
1211
"github.com/spf13/viper"
1312
"github.com/zoobc/zoobc-core/common/crypto"
@@ -21,7 +20,7 @@ import (
2120
func main() {
2221
var apiRPCPort int
2322
if err := util.LoadConfig("../../../resource", "config", "toml"); err != nil {
24-
logrus.Fatal(err)
23+
log.Fatal(err)
2524
} else {
2625
apiRPCPort = viper.GetInt("apiRPCPort")
2726
if apiRPCPort == 0 {

api/client/GetAccountBalance/client.go

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ import (
44
"context"
55
"fmt"
66

7-
"github.com/sirupsen/logrus"
87
log "github.com/sirupsen/logrus"
98
"github.com/spf13/viper"
109
rpc_model "github.com/zoobc/zoobc-core/common/model"
@@ -16,7 +15,7 @@ import (
1615
func main() {
1716
var apiRPCPort int
1817
if err := util.LoadConfig("../../../resource", "config", "toml"); err != nil {
19-
logrus.Fatal(err)
18+
log.Fatal(err)
2019
} else {
2120
apiRPCPort = viper.GetInt("apiRPCPort")
2221
if apiRPCPort == 0 {
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"log"
7+
8+
"github.com/sirupsen/logrus"
9+
"github.com/spf13/viper"
10+
"github.com/zoobc/zoobc-core/common/model"
11+
rpcService "github.com/zoobc/zoobc-core/common/service"
12+
"github.com/zoobc/zoobc-core/common/util"
13+
"google.golang.org/grpc"
14+
)
15+
16+
func main() {
17+
var apiRPCPort int
18+
if err := util.LoadConfig("../../../resource", "config", "toml"); err != nil {
19+
logrus.Fatal(err)
20+
} else {
21+
apiRPCPort = viper.GetInt("apiRPCPort")
22+
}
23+
24+
conn, err := grpc.Dial(fmt.Sprintf(":%d", apiRPCPort), grpc.WithInsecure())
25+
if err != nil {
26+
log.Fatalf("did not connect: %s", err)
27+
}
28+
defer conn.Close()
29+
30+
c := rpcService.NewAccountDatasetServiceClient(conn)
31+
response, err := c.GetAccountDataset(context.Background(), &model.GetAccountDatasetRequest{
32+
RecipientAccountAddress: "H1ftvv3n6CF5NDzdjmZKLRrBg6yPKHXpmatVUhQ5NWYx",
33+
})
34+
if err != nil {
35+
log.Fatalf("error calling grpc GetAccountDatasets: %s", err.Error())
36+
}
37+
log.Printf("response from remote rpc_service.GetTransactions(): %s", response)
38+
}
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"log"
7+
8+
"github.com/sirupsen/logrus"
9+
"github.com/spf13/viper"
10+
"github.com/zoobc/zoobc-core/common/model"
11+
rpcService "github.com/zoobc/zoobc-core/common/service"
12+
"github.com/zoobc/zoobc-core/common/util"
13+
"google.golang.org/grpc"
14+
)
15+
16+
func main() {
17+
var apiRPCPort int
18+
if err := util.LoadConfig("../../../resource", "config", "toml"); err != nil {
19+
logrus.Fatal(err)
20+
} else {
21+
apiRPCPort = viper.GetInt("apiRPCPort")
22+
}
23+
24+
conn, err := grpc.Dial(fmt.Sprintf(":%d", apiRPCPort), grpc.WithInsecure())
25+
if err != nil {
26+
log.Fatalf("did not connect: %s", err)
27+
}
28+
defer conn.Close()
29+
30+
c := rpcService.NewAccountDatasetServiceClient(conn)
31+
response, err := c.GetAccountDatasets(context.Background(), &model.GetAccountDatasetsRequest{
32+
SetterAccountAddress: "HlZLh3VcnNlvByWoAzXOQ2jAlwFOiyO9_njI3oq5Ygha",
33+
RecipientAccountAddress: "H1ftvv3n6CF5NDzdjmZKLRrBg6yPKHXpmatVUhQ5NWYx",
34+
})
35+
if err != nil {
36+
log.Fatalf("error calling grpc GetAccountDatasets: %s", err.Error())
37+
}
38+
log.Printf("response from remote rpc_service.GetTransactions(): %s", response)
39+
}

0 commit comments

Comments
 (0)