Skip to content

add more info in log #713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Mar 31, 2020
Merged

add more info in log #713

merged 2 commits into from
Mar 31, 2020

Conversation

capt4ce
Copy link
Contributor

@capt4ce capt4ce commented Mar 31, 2020

Description

Adding more info to ease analyzing stuck node.

Breakdown

  • Adding more info of cumulative difficulty comparison

Reference Issue

Close -

@capt4ce capt4ce self-assigned this Mar 31, 2020
iltoga
iltoga approved these changes Mar 31, 2020
View reviewed changes
Copy link
Contributor

@iltoga iltoga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@sukrawidhyawan sukrawidhyawan merged commit 45d7392 into develop Mar 31, 2020
andy-shi88 added a commit that referenced this pull request Apr 9, 2020
@astaphobia @iltoga
@andy-shi88 @sg2bcz @sukrawidhyawan @capt4ce @nawikart @KevinH2810
Snapshot, Multisig, Escrow Approval, Fee Model, Bug Fixes (#737)
c137b1b 
* Code Refactoring (#587)

* Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths

* Get escrow transactions with filter fields and pagination (#590)

* get escrow transactions with filter fields and pagination

* get escrow transactions functional test

* #588 decouple published receipt and participation score from block main service (#591)

* 592 fee model (#595)

* #592 add fee model to action type

* #592 add fee model to action type

* #592 ignore package-interface naming lint rule

* #592 update pb file of transaction : add model + service for getTxMinimumFee api

* #592 add handler for calculate minimum fee api

* #592 fix GetTransactionMinimumFee request

* #592 fix wrong type

* #592 add api client for testing get minimum fee

* 578 snapshot file creation (#593)

* Logic for snapshot file creation

* Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain)
* Add ugorji/go/codec package to add CBOR serialization support

* Added Logic for snapshot file processing (create/download)

* Add new interface and service to abstract snapshot file download
* Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks

* Added business logic for snapshots

* Add new service FileService to abstract (and make testable) methods relative to file manipulation

* Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods)
* Move GetFileNameFromHash and GetHashFromFileName into fileservice
* Add query and service handler to retrieve account balances for snapshots
* Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP)
* Add initial logic to create a snapshot file containing all (only last state) account balances
* TODO: increase code coverage (few methods are tested so far)

* Bug fix

* Add EscrowTransactionQuery to GetDerivedQueries

* Added logic for generating snapshot files

* Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService
* Add test case for NewSnapshotFile and new queries

* Finalised logic for generating snapshot files (first iteration only has single file per snapshot)

* Add all relevant queries to get snapshot's payload
* Add method to parse and import snapshot into db
* Add config parameter for snapshots folder
* Add unit and integration tests

* Added logic for download snapshots

* Add method to parse the spineBlockManifest and download file chunk hashes
* TODO: download file is still mocked

* Uncomment logic previously commented for testing purposes
* Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments

* Add query interface for snapshot-specific queries (SnapshotQuery)
* Add SelectDataForSnapshot interface signature to new interface
* Add GetSnapshotQuery method to get all snapshot queries implementations (in a map)
* Refactor relative query methods in every relevant repository
* Refactor some constant to use time.Duration instead of int type
* Fix unit tests

* Add DESC to order by clauses in snapshot queries
* Refactor GetChainTypeCount to GetChainTypes to make chain types countable
* Remove temporary test function


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)

* 596 multisig skeleton (#597)

* #596 add multisignature transaction body

* #596 add constants for MultiSignature body fields

* #596 add multisignature tx type skeleton

* #596 add multisig tx type enum

* #596 add command tools for generate multisignature tx and parsing any tx

* #596 basic multisignature parsing functionality

* #596 update schema commit

* #596 add signature info

* #596 update signature structure to use map

* #596 adjust cmd tools to new multisig byte structure

* #596 update multisig transaction bytes

* #596 update schema

* 599 snapshot chunk strategy (#605)

* Added proto message for snapshots

* Add SnapshotPayload message representing internal snapshot data structure

* Added logic for snapshot chunk 'basic' strategy

* Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot
* Add SnapshotPayload model to schema, since we are using it in two different packages
* Refactor snapshotmainblockservice to use new snapshot chunk strategy
* Refactor unit and integration tests

* Completing logic for snapshot basic chunk strategy

* Finalize BuildSnapshotFromChunks and its test
* Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service

* Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile

* 600 multisig queries (#607)

* #600 add new managed table for multisig

* #600 update multisig managed table model

* #600 add managed table query for multisig

Co-authored-by: Ariasa Ketut <[email protected]>

* 602 generate multisig address (#609)

* method multisig account address assembler

* command line tool multisig generator command

* set 1.20.0 of golangci-lint version to fixed circleci fail

* no need to validate current block height (#610)

* inhibit spine blocks smithing process during main blocks first download (#608)

* Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started)
* Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService
* Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service
* Add new blocker type P2PNetworkConnectionErr
* Increase timers for checking block download process
* Refactor blockStatusService and implementations to blockTypeStatusService
* Refactor relative code in other services
* Remove cast to int snapshotPayloadHeight in snapshotMainBlockService
* Fix bug in snapshotPayloadHeight computation
* Refactor constructor method for BlockchainSyncService

* Fixed transaction generator affected by fee (#611)

* no need to validate current block height

* fixed transcation generator afftected by fee

* Code Refactoring and Bug Fixing (#613)

* Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded)
* Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status
* Remove unused getter from chainType
* Add new constants for better code readability
* Fix tests

* 601 multisig validation (#615)

* #601 add --post and --post-host to enable post transaction bytes on generated

* #601 add multisig validation

* #601 fix lint

* #601 wrong description in cmd

* Optimize GRPC duration (#616)

* no need to validate current block height

* set index into some tables for speed up latency get a lot of records

* remove unused a line of codes

* GRPC endpoint get single escrow transaction by ID (#620)

* no need to validate current block height

* functionality test get escrow by id

* Download snapshot chunks (#614)

* Change fatalf to errorf/return in goroutine in snapshotService.go
* Add missing return statement for error in snapshotService.go

* Added logic for downloading files from a random peer and Code Refactoring

* Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy
* Add rpc call (client request and server response) to download a file from a random peer
* Add logic to download/parse/verify a file and a snapshot

* Added logic and file download metrics

* Add logic to download a file, server side
* Add metrics to count succeeded/failed snapshot downloads

* Update schema: add Failed files for file download
* Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node
* Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones
* Rename megablockQuery to spineBlockManifestQuery
* Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex
* Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data
* Fix smith process lock when downloading spine and main blocks
* Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing


* Add retry download failed file chunks from different peers
* WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string

* Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them
* Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases)
* Optimize p2p.DownloadFilesFromPeer while writing the test
* Fix code after code review

* fixed bug in caclulating snapshot's height

* Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count)
* Refactor code that used sync.Map in favor of new map struct
* Add util function to calculate a quick (non secure) random number
* Fix uncaught nil in p2pServerHandler

* Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks)
* Add a couple of TODOs for spine block manifest validation

* Fix bug in unitialized model.MapIntBool struct
* Add missing test for GenerateSnapshot method

* Uncommented code for testing

* Uncomment code
* Fix one unit test after fixing the snapshot height (previous commit)

Co-authored-by: stefano galassi <[email protected]>

* 594 add bitcoin signature (#618)

* add bitcoin sig, btcd &  btcutil package

* restructure Ed25519 signature type function

* add cmd to generate bitcoin account

* add signature-type flag in cmd tx generator

* cmd enhace switch case generate account

* return error when sign and verify payload

* cmd generate tx with sender signature type

* adjust general parse transaction bytes

* adjust unit test

* udate cmd readme

* cmd, panic when invalid signature type

* remove unused function

* add generate account in signature

* add signature unit test

* add misisng import package

* update schema

* change parameter type of GenerateAccountFromSeed

Co-authored-by: Ariasa Ketut <[email protected]>

* 629 - Use gomod consistently (#630)

* no need to validate current block height

* update circleci config use go modules instead of dep and update go version to 1.14

* go mod and more cleaning shadow declaration variable

* Revert "GRPC endpoint get single escrow transaction by ID (#620)"

This reverts commit 7824924.

* Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)""

This reverts commit bedbf87.

* revert unused commit

* remove Gopkg.* files

* remove no needed comment

* make sure go modules cache correctly

* change circleci working directory, update readme

* update path of cache

* fixed go modules caches

* 603 multisig apply (#623)

* #603 initialize multisig service layer

* #603 implement apply confirmed

* #603 add sender address in pending transaction

* #603 update multisignature.PendingTransaction schema

* #603 delete multisig tx util

* #603 add sign command to support multisig development

* #603 add --sender-address flag for generating unsigned bytes

* #604 add sender address to unsigned tx

* #603 return raw error from execute transaction to enable sqlite error casting

* #603 complete multisig implementation

* #603 fix tests

* #603 add latest flag to pending transaction

* #603 update pending_transaction status on executed

* #603 add latest flag to pending transaction

* #603 add filter to pending transaction query

* #603 add latest flag to multisig-info and signatureInfo

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx model

* #603 add latest field to all pending tables

* #603 adjust query to handle latest flag

* #603 insert multisig-child transaction to transaction table

* #603 dfix test

* #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig)

* #603 fix example cmd

* #603 update schema

* #603 remove binary

* #603 convert address-Signature to format stringTostring

* #603 fix query inconsistency

* #603 handle unsigned transaction execution status

* #603 revert unnecessary change

* #603 handle submit multisig info execute multiple tx

* #603 fix query inconsistency

* #603 add multisig signature type

* #603 add constant for multisig signature validation type;

* #603 remove debug log

* CI delivery image version (#632)

* fixed invalid config

* change image version to stretch

* remove configuration for test purposes

* Ci delivery image version (#633)

* fixed typo

* 604 api rate limit (#625)

* add rate limiting feature for API

* fix unclosed rows

* change MaxAPIRequestPerSecond from int32 to uint32

* add download process metrics tracker

* add go sum

* 621 - Respecting  escrow expiration on Push Block (#626)

* no need to validate current block height

* fixing bug, respecting expiring escrow process on PushBlock

* fixed escrow rollback query and respecting escrow stage before push block process

* fixed escrow unit test result

* has been fixed forgot to close rows

* allow expiring escrow method to control db transaction

* Spine block manifest and spine public keys validation when downloading blocks from peers (#631)

* Bug fixing

* Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers
* Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash)
* Fix tests

* Code Refactoring

* Refactored method's name from GetPayloadHash to GetPayloadHashAndLength

* Added logic to validate block's payload on block push

* Add ValidatePayloadHash logic to blockMainService and blockSpineService
* Fix tests

* Add logic to validate spine block manifest against its relative spine block of reference

* Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp)
* Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network
* Add unit tests for new methods

* Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation

# Conflicts:
#	core/service/blockMainService.go

* Code Refactoring

* Update genesis block ID after changes to Generate block (added published receipts to payload length)
* Add unit test for validateblockhash

* Code Refactoring

* Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods

* Code Refactoring

* Removed unnecessary comment

* Bug fixing

* Remove fmt.println probably left from some internal test

* Bug fixing

* Update comment in query

* Code Optimization

* Optimize if statement by reversing conditions (courtesy of @ali)

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#636)

* Fixed bug in download snapshot process causing segfault

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#639)

* Fix bug in pendingSignatureQuery rollback function
* Remove unused function from transactionQuery

Co-authored-by: stefano galassi <[email protected]>

* Escrow query test fail fixed (#640)

* no need to validate current block height

* escrow query test fail fixed

* Bug Fixing (#642)

* Fix bug in unchecked map key leading to panic in block smith process

* Escrow query test fail fixed (#647)

* no need to validate current block height

* escrow query test fail fixed

* fixed escrow map params need to sorting before compare

* 634 multisig api (#645)

* #634 init multisig api

* #634 update multisig request model

* #634 update get pending tx detail response

* #634 update get pending tx detail response

* #634 initiate multisig client API

* #634 fix bug: wrong empty signature checks

* #634 remove built binary from repository

* #634 add multisig api tests

* #634 revert unnecessary changes

* #634 update go mod

* #634 initiate health check service and model

* #646 add health check endpoint

* #634 update schema

* #634 use getter instead of accessing field directly

* #634 add stricter param check

* #634 fix pagination

* #634 remove duplicate code

Co-authored-by: Ariasa Ketut <[email protected]>

* 634 multisig api (#653)



* #634 add default field to order field param

Co-authored-by: Ariasa Ketut <[email protected]>

* #651 fix multisig_address signature validation (#652)

* #651 fix multisig_address signature validation

* #651 fix test

Co-authored-by: Ariasa Ketut <[email protected]>

* 656 incorrect account ledger balance changes (#657)

* no need to validate current block height

* account ledger incorrect balance change information fixed

* 654 account ledger multisig (#658)

* #654 add multisig transaction event to account ledger

* #654 add ledger record in multisig transaction

* #654 update schema commit

Co-authored-by: Gede Sukra Widhyawan <[email protected]>

* 660 api get multisig info (#661)

* #660 add new api endpoint proto for getting multisignature info

* #660 implement get multisig info api

* #660 rename getPendingTransactions request

* #660 rename getPendingTransactions request

* #660 update schema commit

* 650 - CMD HD wallet generate account (#668)

* no need to validate current block height

* generate account slip10

* allow fetch from private repo

* fixed invalid format of ci

* fixed set env of machine

* add additional ssh key

* fixed bash condition when check gomod exists

* allowed github connection via token instead of ssh

* remove unused code

* more describe generate account command

* fix uninitialized metrics (#662)

fix uninitialized metrics (#662)

* bitcoin sig: add public key into signature (#670)

* bitcoin sig: add public key into signature

* handle multiple rows in a function (#675)

handle multiple rows in a function (#675)

* SelectReceipt Limit (#676)

SelectReceipt Limit (#676)

* Circleci GitHub token (#677)


* token as env variable

* 628 apply snapshot (#667)

* added alpha nodes genesis.go

* Rollback mempool - Restoring mempool transactions backup (#419)

* initiate restoreMempoolsBackup func

* finished mempool backup process and add loggger into processfork struct

* add logger into blockPopper stuff

* 420 get scramble nodes enhance (#423)

* #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now

* #420 move some scrambling function to util to reduce fetching function call frequency

* #420 add block_height to merkle tree table

* #420 fix query to fetch receipt, it had wrong block height scope on the query

* #420 handle error

* #420 add receipt validation

* #420 build scrambled node at height x

* #420 adjust constructor update

* #420 move receipt service instantiation to after node registration service

* #420 move scramble node model outside, it is used by other service as well : receipt service

* #420 validate receipt before storing to database

* #420 use bytes.equal for better comparison performance

* #420 rename memoized scramble nodes

* #420 simplify loop

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted broken query and kept changes from newer pr

* 432 revert participation score query (#437)

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted query

* reverted broken participation score query

* change a few configurations special for alpha (#434)

* change a few configurations special for alpha

* updated config.toml

* added (peer) port to genesis nodes addresses

* fixed test

* updated production values for some constants

* updated log levels for testing alpha nodes

* reset default log levels

* increased PriorityStrategyBuildScrambleNodesGap to 40

* fixed unwanted diffrences between develop and alpha branches

* fixed unit tests for alpha

* fixed unit test for mempoolCoreService

* Fixed unit test

* Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks

* switch log level for debugging purpose

* debug: process fork warn to errorf log

* revert: revert log levels

* keep track of p2p go routine metrics (#569)

* fix shadowed error message

* Resolved merge conflicts

* Code refactoring

* Updated mainchain genesis id

* Code Refactoring

* Update main block genesis ID

* Updated constant

* Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails

* Update golang.org/x/crypto/ed25519 to crypto/ed25519

* Reverted crypto/ed25519 to golang.org/x/crypto/ed25519

* Fixed bug in download snapshot process causing segfault

* Adding business logic

* Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo
* Add logic to snapshot service to include multisig data in snapshots


* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries

* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries
* Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported)
* Add logic to main.go to import snapshot after downloading it

* Refactor IsSnapshotHeight with simplified logic

* Refactor code to be aligned with develop branch instead of alpha

* Refactor SelectDataForSnapshot queries to use rollback-like logic

* Refactor SelectDataForSnapshot queries to use rollback-like logic
* Write unit tests for new queries

* Fix wrong table nale in new block query

* Fix wrong table nale in new block query

* Fix do not allow to delete genesis block when trimming block table

* Revert configX.toml log level to "fatal", "error", "panic"

* #628 add checks if all blocksmith time has expired

* #628 move push block to anonymous function

* Add rollback to snapshot height to set latest = 1 to all latest entitites

* Fixed SelectDataForSnapshot query in accountDataset repo

* Fix missing for loop break conditions in ScanBlockPool
* Add function to log running binary file checksum at node startup

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: stefano galassi <[email protected]>

* 672 makefile binary built (#678)

* #672 ignore release folder

* #672 initialize makefile

* #672 add github token example file

* #672 move xgo to zoobc repository

* #672 add makefile usage

* #672 add cc for cmd

* #672 add go-fmt and separate golangcilint version

* #672 move test to makefile command

* #672 fix gopath

* #672 update readme for github.token usage

* #672 print information for step ran

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* 665-Enhance CMD Database Init (#680)

* cmd: put init db only where is needed

* cmd: add commad to generate proof of ownership

* cmd: add readme rollback

* cmd: fixing remove node reference func

* handle nil value (#682)

* throw error on over limit get blocks (#684)

* throw error on over limit get blocks

* move limit validation to handler layer

* remove pending tx status change, it has been updated in apply confirmed (#685)

* 686: Cmd, enhance generate account (#687)

* cmd: enhance generate account

* add get public key using slip10

* 655 escrow receipt approval (#688)

* no need to validate current block height

* account dataset escrow approval and integration

* fullfill send money validation test cases

* to be gentle no need to check the account dataset property

* create specific method of query for account dataset

* getHostInfo unit testing (#689)

* getHostInfo unit testing
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* adding database status metrics (#693)

adding database status metrics (#693)

* Bug Fixing snapshot selection queries and enhancing rollback queries (#696)

* Bug Fixing

* Fix snapshot selection queries
* Enhance rollback queries
* Enhance db tables by adding composite primary keys where possible
* Fix bug in rollback and snapshot selection queries

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 681 disable smithing non registered node (#694)

* #681 add zero participation score blocker

* #681 inject blockchainStatusService to noderegistration service

* #681 add flag to say if a node is blocksmith or not

* #681 update main.go with new struct constructor

* #681 only update the blocksmith status when detect zero participation score

* #681 compare with bytes.equal()

* unit testing for GetBlockQueue (#698)

unit testing for GetBlockQueue (#698)

* Bug Fixing (#706)

* Fix breaking bug in rollback query
* Refactor spinePublicKey rollback query to comply to new code logic

* 691 grpc account dataset (#699)

* no need to validate current block height

* rename accountDataset to plural name

* account dataset grpc service list and single record

* grpc client example

* move params checkup to handler package

* remove unnecessary test cases

* technical error message passed

* 690 empty block smith (#703)

* #690 add empty flag to enable empty block generate

* #690 add limit for skipped block smith

* #690 set the skipped limit back to 15

* #690 remove unimplemented test

* #690 fix return arguments

* #690 remove unnecessary variable initialization

* #690 make use of default int64 value

* #690 smaller skipped blocksmith constant so we can see the effect on dev net

* adding function for HD wallet testing (#683)

* adding function for HD wallet testing

* P2P: Add unit test on server service (#707)

* p2p: add unit test p2p server service

* enhance Err format

* change GetPeerInfoResponse field name

* p2p: update unit test

* update schema

Co-authored-by: Andy Shi <[email protected]>

* added skippedblocksmith to snapshot (generate/apply) (#710)



* Add SkippedBlocksmith to snapshot proto model

* Added Logic for snapshots

* Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository

* Added Logic for snapshots

* Add logic to include skippedblocksmith records to snapshots
* Add logic to import skippedblocksmith records from snapshots into db (apply snapshot)
* Fix unit tests

* Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot

Co-authored-by: stefano galassi <[email protected]>

* Adding runtime cpu profiling for core node (#712)

* Adding Application Logic
* Add configuration parameter for profiling
* Add runtime flag to enable profiling
* Add logic to run cpu profiling
* Update configuration parameter for profiling
* Add logic to enable memory profiling
* Update documentation (readme.md): see 'run' paragraph

* add more info in log (#713)

* fixed inconsistance block height on escrowed transaction (#717)

* Code Refactoring (#716)

* Use net/http/pprof package that serves the full profiling api via http
* Enable profiling api only when running app with -cpu-profile flag (see updated readme)
* Refactor the way monitoring server is instantiated: not using default http hanlder anymore
* Update readme.md

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#719)

* Refactor block query to select a range of blocsk instad of a single one for snapshots
* Refactor service logic to add more blocks (720) to snapshots

* Cleaning redundant function (#714)

* adding function for HD wallet testing

* cleaning some unused variables

* remove unused const and redundant function on generator

* Bug Fixing (#724)

* Fix pprof http server listening only on localhost

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#728)

* Optimize function GenerateTxRemoveNode

Co-authored-by: stefano galassi <[email protected]>

* 722 blockchain orchestrator (#726)

* add BlockchainOrchestrator to control the chains

* 709 Smithing All Blocksmith Skipped (#725)

* #709 add smithing pending blocker

* #709 remove smith time from blcoksmith model

* #709 update smithing process: recycle blocksmith when all blocksmith time are passed

* #709 separate smith time constants between spinechain and mainchain

* #709 inject spinechain blocksmith service

* #709 update metric smithtime to smith index

* #709 cover new smithing process unit tests

* #709 fix comment on blocksmith strategy & update fake smith function signature

* add index on block table and update sql connnection number (#729)

add index on block table and update sql connnection number (#729)

* scale down the common block jump (#731)

scale down the common block jump (#731)

* 721 - Remove Account Dataset Expiration (#732)

* accountDatasetsQuery should a plural name

* update affected services and more about account_dataset stuff

* fixing unit test of NewSnapshotFile integration

* escrow validation timeout limit exceeded

* Add node registry partial history to snapshots (#734)

* Code Refactoring

* Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process
* Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter
* Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve:
  * latest state from block 0 to 'fromHeight'
  * full history fromHeight to toHeight
  * merge and remove duplicate ids in first selection
* Remove unused constant LinkedReceiptBlocksLimit

Co-authored-by: stefano galassi <[email protected]>

* making the download process less restrictive to failures in catching  up with peers (#735)

* making the download process less restrictive to failures in catching up with peers

* fix the way of tracking errors

* fix the way of keeping track the error

* TestBlockIncompleteQueueService_AddTransaction (#720)

* TestBlockIncompleteQueueService_AddTransaction

* add succes case

* fixed test case spine generate genesis block

Co-authored-by: iltoga <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Kevin Hadi <[email protected]>
andy-shi88 added a commit that referenced this pull request Apr 15, 2020
@astaphobia @iltoga
@andy-shi88 @sg2bcz @sukrawidhyawan @capt4ce @nawikart @KevinH2810
Bug fixes, golangci-lint update version (#754)
8d632d7 
* Code Refactoring (#587)

* Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths

* Get escrow transactions with filter fields and pagination (#590)

* get escrow transactions with filter fields and pagination

* get escrow transactions functional test

* #588 decouple published receipt and participation score from block main service (#591)

* 592 fee model (#595)

* #592 add fee model to action type

* #592 add fee model to action type

* #592 ignore package-interface naming lint rule

* #592 update pb file of transaction : add model + service for getTxMinimumFee api

* #592 add handler for calculate minimum fee api

* #592 fix GetTransactionMinimumFee request

* #592 fix wrong type

* #592 add api client for testing get minimum fee

* 578 snapshot file creation (#593)

* Logic for snapshot file creation

* Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain)
* Add ugorji/go/codec package to add CBOR serialization support

* Added Logic for snapshot file processing (create/download)

* Add new interface and service to abstract snapshot file download
* Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks

* Added business logic for snapshots

* Add new service FileService to abstract (and make testable) methods relative to file manipulation

* Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods)
* Move GetFileNameFromHash and GetHashFromFileName into fileservice
* Add query and service handler to retrieve account balances for snapshots
* Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP)
* Add initial logic to create a snapshot file containing all (only last state) account balances
* TODO: increase code coverage (few methods are tested so far)

* Bug fix

* Add EscrowTransactionQuery to GetDerivedQueries

* Added logic for generating snapshot files

* Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService
* Add test case for NewSnapshotFile and new queries

* Finalised logic for generating snapshot files (first iteration only has single file per snapshot)

* Add all relevant queries to get snapshot's payload
* Add method to parse and import snapshot into db
* Add config parameter for snapshots folder
* Add unit and integration tests

* Added logic for download snapshots

* Add method to parse the spineBlockManifest and download file chunk hashes
* TODO: download file is still mocked

* Uncomment logic previously commented for testing purposes
* Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments

* Add query interface for snapshot-specific queries (SnapshotQuery)
* Add SelectDataForSnapshot interface signature to new interface
* Add GetSnapshotQuery method to get all snapshot queries implementations (in a map)
* Refactor relative query methods in every relevant repository
* Refactor some constant to use time.Duration instead of int type
* Fix unit tests

* Add DESC to order by clauses in snapshot queries
* Refactor GetChainTypeCount to GetChainTypes to make chain types countable
* Remove temporary test function


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)

* 596 multisig skeleton (#597)

* #596 add multisignature transaction body

* #596 add constants for MultiSignature body fields

* #596 add multisignature tx type skeleton

* #596 add multisig tx type enum

* #596 add command tools for generate multisignature tx and parsing any tx

* #596 basic multisignature parsing functionality

* #596 update schema commit

* #596 add signature info

* #596 update signature structure to use map

* #596 adjust cmd tools to new multisig byte structure

* #596 update multisig transaction bytes

* #596 update schema

* 599 snapshot chunk strategy (#605)

* Added proto message for snapshots

* Add SnapshotPayload message representing internal snapshot data structure

* Added logic for snapshot chunk 'basic' strategy

* Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot
* Add SnapshotPayload model to schema, since we are using it in two different packages
* Refactor snapshotmainblockservice to use new snapshot chunk strategy
* Refactor unit and integration tests

* Completing logic for snapshot basic chunk strategy

* Finalize BuildSnapshotFromChunks and its test
* Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service

* Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile

* 600 multisig queries (#607)

* #600 add new managed table for multisig

* #600 update multisig managed table model

* #600 add managed table query for multisig

Co-authored-by: Ariasa Ketut <[email protected]>

* 602 generate multisig address (#609)

* method multisig account address assembler

* command line tool multisig generator command

* set 1.20.0 of golangci-lint version to fixed circleci fail

* no need to validate current block height (#610)

* inhibit spine blocks smithing process during main blocks first download (#608)

* Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started)
* Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService
* Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service
* Add new blocker type P2PNetworkConnectionErr
* Increase timers for checking block download process
* Refactor blockStatusService and implementations to blockTypeStatusService
* Refactor relative code in other services
* Remove cast to int snapshotPayloadHeight in snapshotMainBlockService
* Fix bug in snapshotPayloadHeight computation
* Refactor constructor method for BlockchainSyncService

* Fixed transaction generator affected by fee (#611)

* no need to validate current block height

* fixed transcation generator afftected by fee

* Code Refactoring and Bug Fixing (#613)

* Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded)
* Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status
* Remove unused getter from chainType
* Add new constants for better code readability
* Fix tests

* 601 multisig validation (#615)

* #601 add --post and --post-host to enable post transaction bytes on generated

* #601 add multisig validation

* #601 fix lint

* #601 wrong description in cmd

* Optimize GRPC duration (#616)

* no need to validate current block height

* set index into some tables for speed up latency get a lot of records

* remove unused a line of codes

* GRPC endpoint get single escrow transaction by ID (#620)

* no need to validate current block height

* functionality test get escrow by id

* Download snapshot chunks (#614)

* Change fatalf to errorf/return in goroutine in snapshotService.go
* Add missing return statement for error in snapshotService.go

* Added logic for downloading files from a random peer and Code Refactoring

* Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy
* Add rpc call (client request and server response) to download a file from a random peer
* Add logic to download/parse/verify a file and a snapshot

* Added logic and file download metrics

* Add logic to download a file, server side
* Add metrics to count succeeded/failed snapshot downloads

* Update schema: add Failed files for file download
* Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node
* Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones
* Rename megablockQuery to spineBlockManifestQuery
* Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex
* Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data
* Fix smith process lock when downloading spine and main blocks
* Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing


* Add retry download failed file chunks from different peers
* WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string

* Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them
* Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases)
* Optimize p2p.DownloadFilesFromPeer while writing the test
* Fix code after code review

* fixed bug in caclulating snapshot's height

* Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count)
* Refactor code that used sync.Map in favor of new map struct
* Add util function to calculate a quick (non secure) random number
* Fix uncaught nil in p2pServerHandler

* Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks)
* Add a couple of TODOs for spine block manifest validation

* Fix bug in unitialized model.MapIntBool struct
* Add missing test for GenerateSnapshot method

* Uncommented code for testing

* Uncomment code
* Fix one unit test after fixing the snapshot height (previous commit)

Co-authored-by: stefano galassi <[email protected]>

* 594 add bitcoin signature (#618)

* add bitcoin sig, btcd &  btcutil package

* restructure Ed25519 signature type function

* add cmd to generate bitcoin account

* add signature-type flag in cmd tx generator

* cmd enhace switch case generate account

* return error when sign and verify payload

* cmd generate tx with sender signature type

* adjust general parse transaction bytes

* adjust unit test

* udate cmd readme

* cmd, panic when invalid signature type

* remove unused function

* add generate account in signature

* add signature unit test

* add misisng import package

* update schema

* change parameter type of GenerateAccountFromSeed

Co-authored-by: Ariasa Ketut <[email protected]>

* 629 - Use gomod consistently (#630)

* no need to validate current block height

* update circleci config use go modules instead of dep and update go version to 1.14

* go mod and more cleaning shadow declaration variable

* Revert "GRPC endpoint get single escrow transaction by ID (#620)"

This reverts commit 7824924.

* Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)""

This reverts commit bedbf87.

* revert unused commit

* remove Gopkg.* files

* remove no needed comment

* make sure go modules cache correctly

* change circleci working directory, update readme

* update path of cache

* fixed go modules caches

* 603 multisig apply (#623)

* #603 initialize multisig service layer

* #603 implement apply confirmed

* #603 add sender address in pending transaction

* #603 update multisignature.PendingTransaction schema

* #603 delete multisig tx util

* #603 add sign command to support multisig development

* #603 add --sender-address flag for generating unsigned bytes

* #604 add sender address to unsigned tx

* #603 return raw error from execute transaction to enable sqlite error casting

* #603 complete multisig implementation

* #603 fix tests

* #603 add latest flag to pending transaction

* #603 update pending_transaction status on executed

* #603 add latest flag to pending transaction

* #603 add filter to pending transaction query

* #603 add latest flag to multisig-info and signatureInfo

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx model

* #603 add latest field to all pending tables

* #603 adjust query to handle latest flag

* #603 insert multisig-child transaction to transaction table

* #603 dfix test

* #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig)

* #603 fix example cmd

* #603 update schema

* #603 remove binary

* #603 convert address-Signature to format stringTostring

* #603 fix query inconsistency

* #603 handle unsigned transaction execution status

* #603 revert unnecessary change

* #603 handle submit multisig info execute multiple tx

* #603 fix query inconsistency

* #603 add multisig signature type

* #603 add constant for multisig signature validation type;

* #603 remove debug log

* CI delivery image version (#632)

* fixed invalid config

* change image version to stretch

* remove configuration for test purposes

* Ci delivery image version (#633)

* fixed typo

* 604 api rate limit (#625)

* add rate limiting feature for API

* fix unclosed rows

* change MaxAPIRequestPerSecond from int32 to uint32

* add download process metrics tracker

* add go sum

* 621 - Respecting  escrow expiration on Push Block (#626)

* no need to validate current block height

* fixing bug, respecting expiring escrow process on PushBlock

* fixed escrow rollback query and respecting escrow stage before push block process

* fixed escrow unit test result

* has been fixed forgot to close rows

* allow expiring escrow method to control db transaction

* Spine block manifest and spine public keys validation when downloading blocks from peers (#631)

* Bug fixing

* Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers
* Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash)
* Fix tests

* Code Refactoring

* Refactored method's name from GetPayloadHash to GetPayloadHashAndLength

* Added logic to validate block's payload on block push

* Add ValidatePayloadHash logic to blockMainService and blockSpineService
* Fix tests

* Add logic to validate spine block manifest against its relative spine block of reference

* Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp)
* Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network
* Add unit tests for new methods

* Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation

# Conflicts:
#	core/service/blockMainService.go

* Code Refactoring

* Update genesis block ID after changes to Generate block (added published receipts to payload length)
* Add unit test for validateblockhash

* Code Refactoring

* Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods

* Code Refactoring

* Removed unnecessary comment

* Bug fixing

* Remove fmt.println probably left from some internal test

* Bug fixing

* Update comment in query

* Code Optimization

* Optimize if statement by reversing conditions (courtesy of @ali)

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#636)

* Fixed bug in download snapshot process causing segfault

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#639)

* Fix bug in pendingSignatureQuery rollback function
* Remove unused function from transactionQuery

Co-authored-by: stefano galassi <[email protected]>

* Escrow query test fail fixed (#640)

* no need to validate current block height

* escrow query test fail fixed

* Bug Fixing (#642)

* Fix bug in unchecked map key leading to panic in block smith process

* Escrow query test fail fixed (#647)

* no need to validate current block height

* escrow query test fail fixed

* fixed escrow map params need to sorting before compare

* 634 multisig api (#645)

* #634 init multisig api

* #634 update multisig request model

* #634 update get pending tx detail response

* #634 update get pending tx detail response

* #634 initiate multisig client API

* #634 fix bug: wrong empty signature checks

* #634 remove built binary from repository

* #634 add multisig api tests

* #634 revert unnecessary changes

* #634 update go mod

* #634 initiate health check service and model

* #646 add health check endpoint

* #634 update schema

* #634 use getter instead of accessing field directly

* #634 add stricter param check

* #634 fix pagination

* #634 remove duplicate code

Co-authored-by: Ariasa Ketut <[email protected]>

* 634 multisig api (#653)



* #634 add default field to order field param

Co-authored-by: Ariasa Ketut <[email protected]>

* #651 fix multisig_address signature validation (#652)

* #651 fix multisig_address signature validation

* #651 fix test

Co-authored-by: Ariasa Ketut <[email protected]>

* 656 incorrect account ledger balance changes (#657)

* no need to validate current block height

* account ledger incorrect balance change information fixed

* 654 account ledger multisig (#658)

* #654 add multisig transaction event to account ledger

* #654 add ledger record in multisig transaction

* #654 update schema commit

Co-authored-by: Gede Sukra Widhyawan <[email protected]>

* 660 api get multisig info (#661)

* #660 add new api endpoint proto for getting multisignature info

* #660 implement get multisig info api

* #660 rename getPendingTransactions request

* #660 rename getPendingTransactions request

* #660 update schema commit

* 650 - CMD HD wallet generate account (#668)

* no need to validate current block height

* generate account slip10

* allow fetch from private repo

* fixed invalid format of ci

* fixed set env of machine

* add additional ssh key

* fixed bash condition when check gomod exists

* allowed github connection via token instead of ssh

* remove unused code

* more describe generate account command

* fix uninitialized metrics (#662)

fix uninitialized metrics (#662)

* bitcoin sig: add public key into signature (#670)

* bitcoin sig: add public key into signature

* handle multiple rows in a function (#675)

handle multiple rows in a function (#675)

* SelectReceipt Limit (#676)

SelectReceipt Limit (#676)

* Circleci GitHub token (#677)


* token as env variable

* 628 apply snapshot (#667)

* added alpha nodes genesis.go

* Rollback mempool - Restoring mempool transactions backup (#419)

* initiate restoreMempoolsBackup func

* finished mempool backup process and add loggger into processfork struct

* add logger into blockPopper stuff

* 420 get scramble nodes enhance (#423)

* #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now

* #420 move some scrambling function to util to reduce fetching function call frequency

* #420 add block_height to merkle tree table

* #420 fix query to fetch receipt, it had wrong block height scope on the query

* #420 handle error

* #420 add receipt validation

* #420 build scrambled node at height x

* #420 adjust constructor update

* #420 move receipt service instantiation to after node registration service

* #420 move scramble node model outside, it is used by other service as well : receipt service

* #420 validate receipt before storing to database

* #420 use bytes.equal for better comparison performance

* #420 rename memoized scramble nodes

* #420 simplify loop

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted broken query and kept changes from newer pr

* 432 revert participation score query (#437)

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted query

* reverted broken participation score query

* change a few configurations special for alpha (#434)

* change a few configurations special for alpha

* updated config.toml

* added (peer) port to genesis nodes addresses

* fixed test

* updated production values for some constants

* updated log levels for testing alpha nodes

* reset default log levels

* increased PriorityStrategyBuildScrambleNodesGap to 40

* fixed unwanted diffrences between develop and alpha branches

* fixed unit tests for alpha

* fixed unit test for mempoolCoreService

* Fixed unit test

* Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks

* switch log level for debugging purpose

* debug: process fork warn to errorf log

* revert: revert log levels

* keep track of p2p go routine metrics (#569)

* fix shadowed error message

* Resolved merge conflicts

* Code refactoring

* Updated mainchain genesis id

* Code Refactoring

* Update main block genesis ID

* Updated constant

* Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails

* Update golang.org/x/crypto/ed25519 to crypto/ed25519

* Reverted crypto/ed25519 to golang.org/x/crypto/ed25519

* Fixed bug in download snapshot process causing segfault

* Adding business logic

* Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo
* Add logic to snapshot service to include multisig data in snapshots


* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries

* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries
* Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported)
* Add logic to main.go to import snapshot after downloading it

* Refactor IsSnapshotHeight with simplified logic

* Refactor code to be aligned with develop branch instead of alpha

* Refactor SelectDataForSnapshot queries to use rollback-like logic

* Refactor SelectDataForSnapshot queries to use rollback-like logic
* Write unit tests for new queries

* Fix wrong table nale in new block query

* Fix wrong table nale in new block query

* Fix do not allow to delete genesis block when trimming block table

* Revert configX.toml log level to "fatal", "error", "panic"

* #628 add checks if all blocksmith time has expired

* #628 move push block to anonymous function

* Add rollback to snapshot height to set latest = 1 to all latest entitites

* Fixed SelectDataForSnapshot query in accountDataset repo

* Fix missing for loop break conditions in ScanBlockPool
* Add function to log running binary file checksum at node startup

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: stefano galassi <[email protected]>

* 672 makefile binary built (#678)

* #672 ignore release folder

* #672 initialize makefile

* #672 add github token example file

* #672 move xgo to zoobc repository

* #672 add makefile usage

* #672 add cc for cmd

* #672 add go-fmt and separate golangcilint version

* #672 move test to makefile command

* #672 fix gopath

* #672 update readme for github.token usage

* #672 print information for step ran

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* 665-Enhance CMD Database Init (#680)

* cmd: put init db only where is needed

* cmd: add commad to generate proof of ownership

* cmd: add readme rollback

* cmd: fixing remove node reference func

* handle nil value (#682)

* throw error on over limit get blocks (#684)

* throw error on over limit get blocks

* move limit validation to handler layer

* remove pending tx status change, it has been updated in apply confirmed (#685)

* 686: Cmd, enhance generate account (#687)

* cmd: enhance generate account

* add get public key using slip10

* 655 escrow receipt approval (#688)

* no need to validate current block height

* account dataset escrow approval and integration

* fullfill send money validation test cases

* to be gentle no need to check the account dataset property

* create specific method of query for account dataset

* getHostInfo unit testing (#689)

* getHostInfo unit testing
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* adding database status metrics (#693)

adding database status metrics (#693)

* Bug Fixing snapshot selection queries and enhancing rollback queries (#696)

* Bug Fixing

* Fix snapshot selection queries
* Enhance rollback queries
* Enhance db tables by adding composite primary keys where possible
* Fix bug in rollback and snapshot selection queries

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 681 disable smithing non registered node (#694)

* #681 add zero participation score blocker

* #681 inject blockchainStatusService to noderegistration service

* #681 add flag to say if a node is blocksmith or not

* #681 update main.go with new struct constructor

* #681 only update the blocksmith status when detect zero participation score

* #681 compare with bytes.equal()

* unit testing for GetBlockQueue (#698)

unit testing for GetBlockQueue (#698)

* Bug Fixing (#706)

* Fix breaking bug in rollback query
* Refactor spinePublicKey rollback query to comply to new code logic

* 691 grpc account dataset (#699)

* no need to validate current block height

* rename accountDataset to plural name

* account dataset grpc service list and single record

* grpc client example

* move params checkup to handler package

* remove unnecessary test cases

* technical error message passed

* 690 empty block smith (#703)

* #690 add empty flag to enable empty block generate

* #690 add limit for skipped block smith

* #690 set the skipped limit back to 15

* #690 remove unimplemented test

* #690 fix return arguments

* #690 remove unnecessary variable initialization

* #690 make use of default int64 value

* #690 smaller skipped blocksmith constant so we can see the effect on dev net

* adding function for HD wallet testing (#683)

* adding function for HD wallet testing

* P2P: Add unit test on server service (#707)

* p2p: add unit test p2p server service

* enhance Err format

* change GetPeerInfoResponse field name

* p2p: update unit test

* update schema

Co-authored-by: Andy Shi <[email protected]>

* added skippedblocksmith to snapshot (generate/apply) (#710)



* Add SkippedBlocksmith to snapshot proto model

* Added Logic for snapshots

* Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository

* Added Logic for snapshots

* Add logic to include skippedblocksmith records to snapshots
* Add logic to import skippedblocksmith records from snapshots into db (apply snapshot)
* Fix unit tests

* Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot

Co-authored-by: stefano galassi <[email protected]>

* Adding runtime cpu profiling for core node (#712)

* Adding Application Logic
* Add configuration parameter for profiling
* Add runtime flag to enable profiling
* Add logic to run cpu profiling
* Update configuration parameter for profiling
* Add logic to enable memory profiling
* Update documentation (readme.md): see 'run' paragraph

* add more info in log (#713)

* fixed inconsistance block height on escrowed transaction (#717)

* Code Refactoring (#716)

* Use net/http/pprof package that serves the full profiling api via http
* Enable profiling api only when running app with -cpu-profile flag (see updated readme)
* Refactor the way monitoring server is instantiated: not using default http hanlder anymore
* Update readme.md

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#719)

* Refactor block query to select a range of blocsk instad of a single one for snapshots
* Refactor service logic to add more blocks (720) to snapshots

* Cleaning redundant function (#714)

* adding function for HD wallet testing

* cleaning some unused variables

* remove unused const and redundant function on generator

* Bug Fixing (#724)

* Fix pprof http server listening only on localhost

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#728)

* Optimize function GenerateTxRemoveNode

Co-authored-by: stefano galassi <[email protected]>

* 722 blockchain orchestrator (#726)

* add BlockchainOrchestrator to control the chains

* 709 Smithing All Blocksmith Skipped (#725)

* #709 add smithing pending blocker

* #709 remove smith time from blcoksmith model

* #709 update smithing process: recycle blocksmith when all blocksmith time are passed

* #709 separate smith time constants between spinechain and mainchain

* #709 inject spinechain blocksmith service

* #709 update metric smithtime to smith index

* #709 cover new smithing process unit tests

* #709 fix comment on blocksmith strategy & update fake smith function signature

* add index on block table and update sql connnection number (#729)

add index on block table and update sql connnection number (#729)

* scale down the common block jump (#731)

scale down the common block jump (#731)

* 721 - Remove Account Dataset Expiration (#732)

* accountDatasetsQuery should a plural name

* update affected services and more about account_dataset stuff

* fixing unit test of NewSnapshotFile integration

* escrow validation timeout limit exceeded

* Add node registry partial history to snapshots (#734)

* Code Refactoring

* Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process
* Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter
* Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve:
  * latest state from block 0 to 'fromHeight'
  * full history fromHeight to toHeight
  * merge and remove duplicate ids in first selection
* Remove unused constant LinkedReceiptBlocksLimit

Co-authored-by: stefano galassi <[email protected]>

* making the download process less restrictive to failures in catching  up with peers (#735)

* making the download process less restrictive to failures in catching up with peers

* fix the way of tracking errors

* fix the way of keeping track the error

* TestBlockIncompleteQueueService_AddTransaction (#720)

* TestBlockIncompleteQueueService_AddTransaction

* add succes case

* fix return type to use blocker (#742)

* fix return type to use blocker

* add more blocker type

* Update golangci to 1.24.0 (#743)

* update golangci to 1.24

* add reset db variable step

* fixed invalid space

* test circlleci building a branch

* fixes golangci warn and error

* disable ineffassign

* adjust the deployment step

* #740 index receipt fields for better query time (#744)

* #740 index receipt fields for faster query

* #740 verbose message on golangcilint run

* fix error casting (#747)

* fix error casting

* revert p2p error type

* fix blocker usage

* downgrade golangci-lint version (#748)

* Hotfix rows immediate release (#749)

* immediately release rows after converted to model

* add unique index to published_receipt.datum_hash

* fix: wrong column name

* temporary fix, omit the unique

* Unit Testing core service BlockPoolService (#739)

* NewBlockPoolService, GetBlocks, GetBlock, InsertBlock, ClearBlockPool

* add assertion

* gofmt -s -w

* Add optional Params on Sign function (#746)

* enhance sign function & cmd signature

* update readme

* add option params on GenerateSignedTxBytes

* update comment function SignEd25519

* add return err on parseBytesArgs func

* fix type & unused arg

* missing flag ed25519UseSlip10

* handle error from block push and validation (#751)

handle error from block push and validation (#751)

* fix block validation and smith time calculation (#752)

* fix genesis files

* setter and recipient must be the same person (#755)

Co-authored-by: iltoga <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Kevin Hadi <[email protected]>
@astaphobia astaphobia deleted the add-more-info-in-log branch April 15, 2020 01:18
andy-shi88 added a commit that referenced this pull request May 14, 2020
@astaphobia @iltoga
@andy-shi88 @sg2bcz @sukrawidhyawan @capt4ce @nawikart @KevinH2810
Bug fixes and tidying up (#799)
cede23d 
* Code Refactoring (#587)

* Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths

* Get escrow transactions with filter fields and pagination (#590)

* get escrow transactions with filter fields and pagination

* get escrow transactions functional test

* #588 decouple published receipt and participation score from block main service (#591)

* 592 fee model (#595)

* #592 add fee model to action type

* #592 add fee model to action type

* #592 ignore package-interface naming lint rule

* #592 update pb file of transaction : add model + service for getTxMinimumFee api

* #592 add handler for calculate minimum fee api

* #592 fix GetTransactionMinimumFee request

* #592 fix wrong type

* #592 add api client for testing get minimum fee

* 578 snapshot file creation (#593)

* Logic for snapshot file creation

* Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain)
* Add ugorji/go/codec package to add CBOR serialization support

* Added Logic for snapshot file processing (create/download)

* Add new interface and service to abstract snapshot file download
* Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks

* Added business logic for snapshots

* Add new service FileService to abstract (and make testable) methods relative to file manipulation

* Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods)
* Move GetFileNameFromHash and GetHashFromFileName into fileservice
* Add query and service handler to retrieve account balances for snapshots
* Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP)
* Add initial logic to create a snapshot file containing all (only last state) account balances
* TODO: increase code coverage (few methods are tested so far)

* Bug fix

* Add EscrowTransactionQuery to GetDerivedQueries

* Added logic for generating snapshot files

* Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService
* Add test case for NewSnapshotFile and new queries

* Finalised logic for generating snapshot files (first iteration only has single file per snapshot)

* Add all relevant queries to get snapshot's payload
* Add method to parse and import snapshot into db
* Add config parameter for snapshots folder
* Add unit and integration tests

* Added logic for download snapshots

* Add method to parse the spineBlockManifest and download file chunk hashes
* TODO: download file is still mocked

* Uncomment logic previously commented for testing purposes
* Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments

* Add query interface for snapshot-specific queries (SnapshotQuery)
* Add SelectDataForSnapshot interface signature to new interface
* Add GetSnapshotQuery method to get all snapshot queries implementations (in a map)
* Refactor relative query methods in every relevant repository
* Refactor some constant to use time.Duration instead of int type
* Fix unit tests

* Add DESC to order by clauses in snapshot queries
* Refactor GetChainTypeCount to GetChainTypes to make chain types countable
* Remove temporary test function


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)

* 596 multisig skeleton (#597)

* #596 add multisignature transaction body

* #596 add constants for MultiSignature body fields

* #596 add multisignature tx type skeleton

* #596 add multisig tx type enum

* #596 add command tools for generate multisignature tx and parsing any tx

* #596 basic multisignature parsing functionality

* #596 update schema commit

* #596 add signature info

* #596 update signature structure to use map

* #596 adjust cmd tools to new multisig byte structure

* #596 update multisig transaction bytes

* #596 update schema

* 599 snapshot chunk strategy (#605)

* Added proto message for snapshots

* Add SnapshotPayload message representing internal snapshot data structure

* Added logic for snapshot chunk 'basic' strategy

* Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot
* Add SnapshotPayload model to schema, since we are using it in two different packages
* Refactor snapshotmainblockservice to use new snapshot chunk strategy
* Refactor unit and integration tests

* Completing logic for snapshot basic chunk strategy

* Finalize BuildSnapshotFromChunks and its test
* Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service

* Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile

* 600 multisig queries (#607)

* #600 add new managed table for multisig

* #600 update multisig managed table model

* #600 add managed table query for multisig

Co-authored-by: Ariasa Ketut <[email protected]>

* 602 generate multisig address (#609)

* method multisig account address assembler

* command line tool multisig generator command

* set 1.20.0 of golangci-lint version to fixed circleci fail

* no need to validate current block height (#610)

* inhibit spine blocks smithing process during main blocks first download (#608)

* Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started)
* Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService
* Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service
* Add new blocker type P2PNetworkConnectionErr
* Increase timers for checking block download process
* Refactor blockStatusService and implementations to blockTypeStatusService
* Refactor relative code in other services
* Remove cast to int snapshotPayloadHeight in snapshotMainBlockService
* Fix bug in snapshotPayloadHeight computation
* Refactor constructor method for BlockchainSyncService

* Fixed transaction generator affected by fee (#611)

* no need to validate current block height

* fixed transcation generator afftected by fee

* Code Refactoring and Bug Fixing (#613)

* Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded)
* Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status
* Remove unused getter from chainType
* Add new constants for better code readability
* Fix tests

* 601 multisig validation (#615)

* #601 add --post and --post-host to enable post transaction bytes on generated

* #601 add multisig validation

* #601 fix lint

* #601 wrong description in cmd

* Optimize GRPC duration (#616)

* no need to validate current block height

* set index into some tables for speed up latency get a lot of records

* remove unused a line of codes

* GRPC endpoint get single escrow transaction by ID (#620)

* no need to validate current block height

* functionality test get escrow by id

* Download snapshot chunks (#614)

* Change fatalf to errorf/return in goroutine in snapshotService.go
* Add missing return statement for error in snapshotService.go

* Added logic for downloading files from a random peer and Code Refactoring

* Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy
* Add rpc call (client request and server response) to download a file from a random peer
* Add logic to download/parse/verify a file and a snapshot

* Added logic and file download metrics

* Add logic to download a file, server side
* Add metrics to count succeeded/failed snapshot downloads

* Update schema: add Failed files for file download
* Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node
* Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones
* Rename megablockQuery to spineBlockManifestQuery
* Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex
* Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data
* Fix smith process lock when downloading spine and main blocks
* Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing


* Add retry download failed file chunks from different peers
* WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string

* Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them
* Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases)
* Optimize p2p.DownloadFilesFromPeer while writing the test
* Fix code after code review

* fixed bug in caclulating snapshot's height

* Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count)
* Refactor code that used sync.Map in favor of new map struct
* Add util function to calculate a quick (non secure) random number
* Fix uncaught nil in p2pServerHandler

* Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks)
* Add a couple of TODOs for spine block manifest validation

* Fix bug in unitialized model.MapIntBool struct
* Add missing test for GenerateSnapshot method

* Uncommented code for testing

* Uncomment code
* Fix one unit test after fixing the snapshot height (previous commit)

Co-authored-by: stefano galassi <[email protected]>

* 594 add bitcoin signature (#618)

* add bitcoin sig, btcd &  btcutil package

* restructure Ed25519 signature type function

* add cmd to generate bitcoin account

* add signature-type flag in cmd tx generator

* cmd enhace switch case generate account

* return error when sign and verify payload

* cmd generate tx with sender signature type

* adjust general parse transaction bytes

* adjust unit test

* udate cmd readme

* cmd, panic when invalid signature type

* remove unused function

* add generate account in signature

* add signature unit test

* add misisng import package

* update schema

* change parameter type of GenerateAccountFromSeed

Co-authored-by: Ariasa Ketut <[email protected]>

* 629 - Use gomod consistently (#630)

* no need to validate current block height

* update circleci config use go modules instead of dep and update go version to 1.14

* go mod and more cleaning shadow declaration variable

* Revert "GRPC endpoint get single escrow transaction by ID (#620)"

This reverts commit 7824924.

* Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)""

This reverts commit bedbf87.

* revert unused commit

* remove Gopkg.* files

* remove no needed comment

* make sure go modules cache correctly

* change circleci working directory, update readme

* update path of cache

* fixed go modules caches

* 603 multisig apply (#623)

* #603 initialize multisig service layer

* #603 implement apply confirmed

* #603 add sender address in pending transaction

* #603 update multisignature.PendingTransaction schema

* #603 delete multisig tx util

* #603 add sign command to support multisig development

* #603 add --sender-address flag for generating unsigned bytes

* #604 add sender address to unsigned tx

* #603 return raw error from execute transaction to enable sqlite error casting

* #603 complete multisig implementation

* #603 fix tests

* #603 add latest flag to pending transaction

* #603 update pending_transaction status on executed

* #603 add latest flag to pending transaction

* #603 add filter to pending transaction query

* #603 add latest flag to multisig-info and signatureInfo

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx model

* #603 add latest field to all pending tables

* #603 adjust query to handle latest flag

* #603 insert multisig-child transaction to transaction table

* #603 dfix test

* #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig)

* #603 fix example cmd

* #603 update schema

* #603 remove binary

* #603 convert address-Signature to format stringTostring

* #603 fix query inconsistency

* #603 handle unsigned transaction execution status

* #603 revert unnecessary change

* #603 handle submit multisig info execute multiple tx

* #603 fix query inconsistency

* #603 add multisig signature type

* #603 add constant for multisig signature validation type;

* #603 remove debug log

* CI delivery image version (#632)

* fixed invalid config

* change image version to stretch

* remove configuration for test purposes

* Ci delivery image version (#633)

* fixed typo

* 604 api rate limit (#625)

* add rate limiting feature for API

* fix unclosed rows

* change MaxAPIRequestPerSecond from int32 to uint32

* add download process metrics tracker

* add go sum

* 621 - Respecting  escrow expiration on Push Block (#626)

* no need to validate current block height

* fixing bug, respecting expiring escrow process on PushBlock

* fixed escrow rollback query and respecting escrow stage before push block process

* fixed escrow unit test result

* has been fixed forgot to close rows

* allow expiring escrow method to control db transaction

* Spine block manifest and spine public keys validation when downloading blocks from peers (#631)

* Bug fixing

* Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers
* Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash)
* Fix tests

* Code Refactoring

* Refactored method's name from GetPayloadHash to GetPayloadHashAndLength

* Added logic to validate block's payload on block push

* Add ValidatePayloadHash logic to blockMainService and blockSpineService
* Fix tests

* Add logic to validate spine block manifest against its relative spine block of reference

* Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp)
* Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network
* Add unit tests for new methods

* Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation

# Conflicts:
#	core/service/blockMainService.go

* Code Refactoring

* Update genesis block ID after changes to Generate block (added published receipts to payload length)
* Add unit test for validateblockhash

* Code Refactoring

* Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods

* Code Refactoring

* Removed unnecessary comment

* Bug fixing

* Remove fmt.println probably left from some internal test

* Bug fixing

* Update comment in query

* Code Optimization

* Optimize if statement by reversing conditions (courtesy of @ali)

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#636)

* Fixed bug in download snapshot process causing segfault

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#639)

* Fix bug in pendingSignatureQuery rollback function
* Remove unused function from transactionQuery

Co-authored-by: stefano galassi <[email protected]>

* Escrow query test fail fixed (#640)

* no need to validate current block height

* escrow query test fail fixed

* Bug Fixing (#642)

* Fix bug in unchecked map key leading to panic in block smith process

* Escrow query test fail fixed (#647)

* no need to validate current block height

* escrow query test fail fixed

* fixed escrow map params need to sorting before compare

* 634 multisig api (#645)

* #634 init multisig api

* #634 update multisig request model

* #634 update get pending tx detail response

* #634 update get pending tx detail response

* #634 initiate multisig client API

* #634 fix bug: wrong empty signature checks

* #634 remove built binary from repository

* #634 add multisig api tests

* #634 revert unnecessary changes

* #634 update go mod

* #634 initiate health check service and model

* #646 add health check endpoint

* #634 update schema

* #634 use getter instead of accessing field directly

* #634 add stricter param check

* #634 fix pagination

* #634 remove duplicate code

Co-authored-by: Ariasa Ketut <[email protected]>

* 634 multisig api (#653)



* #634 add default field to order field param

Co-authored-by: Ariasa Ketut <[email protected]>

* #651 fix multisig_address signature validation (#652)

* #651 fix multisig_address signature validation

* #651 fix test

Co-authored-by: Ariasa Ketut <[email protected]>

* 656 incorrect account ledger balance changes (#657)

* no need to validate current block height

* account ledger incorrect balance change information fixed

* 654 account ledger multisig (#658)

* #654 add multisig transaction event to account ledger

* #654 add ledger record in multisig transaction

* #654 update schema commit

Co-authored-by: Gede Sukra Widhyawan <[email protected]>

* 660 api get multisig info (#661)

* #660 add new api endpoint proto for getting multisignature info

* #660 implement get multisig info api

* #660 rename getPendingTransactions request

* #660 rename getPendingTransactions request

* #660 update schema commit

* 650 - CMD HD wallet generate account (#668)

* no need to validate current block height

* generate account slip10

* allow fetch from private repo

* fixed invalid format of ci

* fixed set env of machine

* add additional ssh key

* fixed bash condition when check gomod exists

* allowed github connection via token instead of ssh

* remove unused code

* more describe generate account command

* fix uninitialized metrics (#662)

fix uninitialized metrics (#662)

* bitcoin sig: add public key into signature (#670)

* bitcoin sig: add public key into signature

* handle multiple rows in a function (#675)

handle multiple rows in a function (#675)

* SelectReceipt Limit (#676)

SelectReceipt Limit (#676)

* Circleci GitHub token (#677)


* token as env variable

* 628 apply snapshot (#667)

* added alpha nodes genesis.go

* Rollback mempool - Restoring mempool transactions backup (#419)

* initiate restoreMempoolsBackup func

* finished mempool backup process and add loggger into processfork struct

* add logger into blockPopper stuff

* 420 get scramble nodes enhance (#423)

* #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now

* #420 move some scrambling function to util to reduce fetching function call frequency

* #420 add block_height to merkle tree table

* #420 fix query to fetch receipt, it had wrong block height scope on the query

* #420 handle error

* #420 add receipt validation

* #420 build scrambled node at height x

* #420 adjust constructor update

* #420 move receipt service instantiation to after node registration service

* #420 move scramble node model outside, it is used by other service as well : receipt service

* #420 validate receipt before storing to database

* #420 use bytes.equal for better comparison performance

* #420 rename memoized scramble nodes

* #420 simplify loop

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted broken query and kept changes from newer pr

* 432 revert participation score query (#437)

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted query

* reverted broken participation score query

* change a few configurations special for alpha (#434)

* change a few configurations special for alpha

* updated config.toml

* added (peer) port to genesis nodes addresses

* fixed test

* updated production values for some constants

* updated log levels for testing alpha nodes

* reset default log levels

* increased PriorityStrategyBuildScrambleNodesGap to 40

* fixed unwanted diffrences between develop and alpha branches

* fixed unit tests for alpha

* fixed unit test for mempoolCoreService

* Fixed unit test

* Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks

* switch log level for debugging purpose

* debug: process fork warn to errorf log

* revert: revert log levels

* keep track of p2p go routine metrics (#569)

* fix shadowed error message

* Resolved merge conflicts

* Code refactoring

* Updated mainchain genesis id

* Code Refactoring

* Update main block genesis ID

* Updated constant

* Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails

* Update golang.org/x/crypto/ed25519 to crypto/ed25519

* Reverted crypto/ed25519 to golang.org/x/crypto/ed25519

* Fixed bug in download snapshot process causing segfault

* Adding business logic

* Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo
* Add logic to snapshot service to include multisig data in snapshots


* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries

* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries
* Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported)
* Add logic to main.go to import snapshot after downloading it

* Refactor IsSnapshotHeight with simplified logic

* Refactor code to be aligned with develop branch instead of alpha

* Refactor SelectDataForSnapshot queries to use rollback-like logic

* Refactor SelectDataForSnapshot queries to use rollback-like logic
* Write unit tests for new queries

* Fix wrong table nale in new block query

* Fix wrong table nale in new block query

* Fix do not allow to delete genesis block when trimming block table

* Revert configX.toml log level to "fatal", "error", "panic"

* #628 add checks if all blocksmith time has expired

* #628 move push block to anonymous function

* Add rollback to snapshot height to set latest = 1 to all latest entitites

* Fixed SelectDataForSnapshot query in accountDataset repo

* Fix missing for loop break conditions in ScanBlockPool
* Add function to log running binary file checksum at node startup

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: stefano galassi <[email protected]>

* 672 makefile binary built (#678)

* #672 ignore release folder

* #672 initialize makefile

* #672 add github token example file

* #672 move xgo to zoobc repository

* #672 add makefile usage

* #672 add cc for cmd

* #672 add go-fmt and separate golangcilint version

* #672 move test to makefile command

* #672 fix gopath

* #672 update readme for github.token usage

* #672 print information for step ran

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* 665-Enhance CMD Database Init (#680)

* cmd: put init db only where is needed

* cmd: add commad to generate proof of ownership

* cmd: add readme rollback

* cmd: fixing remove node reference func

* handle nil value (#682)

* throw error on over limit get blocks (#684)

* throw error on over limit get blocks

* move limit validation to handler layer

* remove pending tx status change, it has been updated in apply confirmed (#685)

* 686: Cmd, enhance generate account (#687)

* cmd: enhance generate account

* add get public key using slip10

* 655 escrow receipt approval (#688)

* no need to validate current block height

* account dataset escrow approval and integration

* fullfill send money validation test cases

* to be gentle no need to check the account dataset property

* create specific method of query for account dataset

* getHostInfo unit testing (#689)

* getHostInfo unit testing
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* adding database status metrics (#693)

adding database status metrics (#693)

* Bug Fixing snapshot selection queries and enhancing rollback queries (#696)

* Bug Fixing

* Fix snapshot selection queries
* Enhance rollback queries
* Enhance db tables by adding composite primary keys where possible
* Fix bug in rollback and snapshot selection queries

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 681 disable smithing non registered node (#694)

* #681 add zero participation score blocker

* #681 inject blockchainStatusService to noderegistration service

* #681 add flag to say if a node is blocksmith or not

* #681 update main.go with new struct constructor

* #681 only update the blocksmith status when detect zero participation score

* #681 compare with bytes.equal()

* unit testing for GetBlockQueue (#698)

unit testing for GetBlockQueue (#698)

* Bug Fixing (#706)

* Fix breaking bug in rollback query
* Refactor spinePublicKey rollback query to comply to new code logic

* 691 grpc account dataset (#699)

* no need to validate current block height

* rename accountDataset to plural name

* account dataset grpc service list and single record

* grpc client example

* move params checkup to handler package

* remove unnecessary test cases

* technical error message passed

* 690 empty block smith (#703)

* #690 add empty flag to enable empty block generate

* #690 add limit for skipped block smith

* #690 set the skipped limit back to 15

* #690 remove unimplemented test

* #690 fix return arguments

* #690 remove unnecessary variable initialization

* #690 make use of default int64 value

* #690 smaller skipped blocksmith constant so we can see the effect on dev net

* adding function for HD wallet testing (#683)

* adding function for HD wallet testing

* P2P: Add unit test on server service (#707)

* p2p: add unit test p2p server service

* enhance Err format

* change GetPeerInfoResponse field name

* p2p: update unit test

* update schema

Co-authored-by: Andy Shi <[email protected]>

* added skippedblocksmith to snapshot (generate/apply) (#710)



* Add SkippedBlocksmith to snapshot proto model

* Added Logic for snapshots

* Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository

* Added Logic for snapshots

* Add logic to include skippedblocksmith records to snapshots
* Add logic to import skippedblocksmith records from snapshots into db (apply snapshot)
* Fix unit tests

* Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot

Co-authored-by: stefano galassi <[email protected]>

* Adding runtime cpu profiling for core node (#712)

* Adding Application Logic
* Add configuration parameter for profiling
* Add runtime flag to enable profiling
* Add logic to run cpu profiling
* Update configuration parameter for profiling
* Add logic to enable memory profiling
* Update documentation (readme.md): see 'run' paragraph

* add more info in log (#713)

* fixed inconsistance block height on escrowed transaction (#717)

* Code Refactoring (#716)

* Use net/http/pprof package that serves the full profiling api via http
* Enable profiling api only when running app with -cpu-profile flag (see updated readme)
* Refactor the way monitoring server is instantiated: not using default http hanlder anymore
* Update readme.md

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#719)

* Refactor block query to select a range of blocsk instad of a single one for snapshots
* Refactor service logic to add more blocks (720) to snapshots

* Cleaning redundant function (#714)

* adding function for HD wallet testing

* cleaning some unused variables

* remove unused const and redundant function on generator

* Bug Fixing (#724)

* Fix pprof http server listening only on localhost

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#728)

* Optimize function GenerateTxRemoveNode

Co-authored-by: stefano galassi <[email protected]>

* 722 blockchain orchestrator (#726)

* add BlockchainOrchestrator to control the chains

* 709 Smithing All Blocksmith Skipped (#725)

* #709 add smithing pending blocker

* #709 remove smith time from blcoksmith model

* #709 update smithing process: recycle blocksmith when all blocksmith time are passed

* #709 separate smith time constants between spinechain and mainchain

* #709 inject spinechain blocksmith service

* #709 update metric smithtime to smith index

* #709 cover new smithing process unit tests

* #709 fix comment on blocksmith strategy & update fake smith function signature

* add index on block table and update sql connnection number (#729)

add index on block table and update sql connnection number (#729)

* scale down the common block jump (#731)

scale down the common block jump (#731)

* 721 - Remove Account Dataset Expiration (#732)

* accountDatasetsQuery should a plural name

* update affected services and more about account_dataset stuff

* fixing unit test of NewSnapshotFile integration

* escrow validation timeout limit exceeded

* Add node registry partial history to snapshots (#734)

* Code Refactoring

* Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process
* Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter
* Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve:
  * latest state from block 0 to 'fromHeight'
  * full history fromHeight to toHeight
  * merge and remove duplicate ids in first selection
* Remove unused constant LinkedReceiptBlocksLimit

Co-authored-by: stefano galassi <[email protected]>

* making the download process less restrictive to failures in catching  up with peers (#735)

* making the download process less restrictive to failures in catching up with peers

* fix the way of tracking errors

* fix the way of keeping track the error

* TestBlockIncompleteQueueService_AddTransaction (#720)

* TestBlockIncompleteQueueService_AddTransaction

* add succes case

* fix return type to use blocker (#742)

* fix return type to use blocker

* add more blocker type

* Update golangci to 1.24.0 (#743)

* update golangci to 1.24

* add reset db variable step

* fixed invalid space

* test circlleci building a branch

* fixes golangci warn and error

* disable ineffassign

* adjust the deployment step

* #740 index receipt fields for better query time (#744)

* #740 index receipt fields for faster query

* #740 verbose message on golangcilint run

* fix error casting (#747)

* fix error casting

* revert p2p error type

* fix blocker usage

* downgrade golangci-lint version (#748)

* Hotfix rows immediate release (#749)

* immediately release rows after converted to model

* add unique index to published_receipt.datum_hash

* fix: wrong column name

* temporary fix, omit the unique

* Unit Testing core service BlockPoolService (#739)

* NewBlockPoolService, GetBlocks, GetBlock, InsertBlock, ClearBlockPool

* add assertion

* gofmt -s -w

* Add optional Params on Sign function (#746)

* enhance sign function & cmd signature

* update readme

* add option params on GenerateSignedTxBytes

* update comment function SignEd25519

* add return err on parseBytesArgs func

* fix type & unused arg

* missing flag ed25519UseSlip10

* handle error from block push and validation (#751)

handle error from block push and validation (#751)

* fix block validation and smith time calculation (#752)

* setter and recipient must be the same person (#755)

* Fix: P2P - Can't Add UnresolvedPeers  (#757)

* add condition when there are still space in unresolved peers

* remove duplicate removal query (#758)

* Optimize database transaction (#756)

* use ExecuteSelectRow instead

* use QueryRow instead of Query, get single account balance

* use QueryRow instead if only want a single record

* nodeReg.ApplyConfirmed better to use QueryRow

* nodeRegUpdate.ApplyConfirmed better to use QueryRow

* removeNodeReg.ApplyConfirmed better to use QueryRow

* CMD fix and addition (#762)

* add blockheight logging at the end of applying a snapshot

* fix rollback cmd method to use the parameter passed

* add cmd methods for getting scrambledNodes and priorityPeers

* rename folder of scrambled nodes

* Use QueryRow instead of Query when getting single record (#764)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix query and the unit tests (#765)

fix query and the unit tests (#765)

* Confirm the state of download finish if the whole network stuck (#770)

* confirm the state of download finish if the whole network stuck

* reset the firstDownloadCounter when finding out they are not stuck

* reduce the validation limit constant

* Unit Test Transaction Core Service (#767)

* UndoApplyUnconfirmedTransaction

* ApplyConfirmedTransaction

* ApplyUnconfirmedTransaction

* ValidateTransaction

* remove underscores in Go names

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* GetBlockByHeight: Query row instead of Query (#768)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

* GetBlockByHeight better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix finish download (#771)

fix finish download (#771)

* reset db automatically and also snapshots stuff (#772)

* reset db automatically and also snapshots stuff

* *_kv would remove also

* snapshots suffix

* Fix manifest rollback (#773)

* add spineblock height field

* rename spine block manifest column and add indesx on block height column

* rename spine_block_manifest column field

* move spine_block_manifest to spinechain derived query category

* clean snapshot file on rollbacked

* add delete snapshot file function

* update tests

* remove ErrNoRows check as that error only occur on sql.Row not sql.Rows

* inject snapshot block service to spine block service

* snapshot manifest field rename

* fix new rollback query for manifest

* use parse instead of splitting chunks manually

* log the deletion error

* handle error

* update schema

* add extra comment for testing

* move file deletion to after db commit and as goroutine

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* #777 update validate block function signature : remove timestamp (#779)

* #777 update validate block function signature : remove timestamp

* move the validation of download finish if the block height is 0 (#781)

move the validation of download finish if the block height is 0 (#781)

* Bug limit receipt selection when select by root 774 (#776)

* versi 1

* small edit

* update mock ExecuteSelect

* make sure the query is similar as mock

* make it simpler

* merge schema

Co-authored-by: Nawi Kartini <[email protected]>

* 782 missing blocks fix (#783)

* #782 validate block in scanBlockPool thread

* #782 fix wrong function signature causing interface not implemented

* #782 move chain lock to beginning of ScanBlockPool function

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* Getting own public ip (#784)

* get own public ip

* allowing to get public ip via checkip.dyndns.org

* make sure the parsed ip is valid ip address

* IPUtil methods

* Fixing error nil pointer when accessing sql.Tx Executor (#786)

* add missing err checker on validate mempool

* fixing racing unlocking mutex & making sql.Tx nil

* Sort poppedBlocks by height ascending (#787)

* sort poppedBlocks by height desc

* fulfill test cases of PoppedoffBlocks

* sort ascending popped off blocks

* fulfill spine PopOffToBlock test cases

* #792 replace spendable with balance (#793)

* Genesis generate command: Code Refactoring and adding new feature to facilitate deployments with nomad/consul (#797)

* Code Refactoring and adding new feature

* Refactor command code to comply to new standards (see other commands)
* Add feature to generate consulKvInit.sh script from a template and a the newly generated genesis

* Bug fixing

* Limiter must be 2*MinRollbackHeight (#795)

* limiter must be 2*MinRollbackHeight

* 789 multisig pending transaction Apply-Flow (#798)

* #789 update pending_transaction query by hash to have multiple status filter

* #789 update flow of pending_transaction execution

* Tech: GetAccountBalances #785 (#788)

* first commit

* check num of addresses

* fix

* fix

* fix

* fix the unit testing

* use caseQuery instead of looping

* fix caseQuery.In

* fix unit testing

* move validation to handler

* fix unit test part 1

* auto update from proto

* accountBalance

* change to status.Error

* change to plural

* refine according PR comment

* fix

* fix unit test part 1

* remove commented line

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* fix the status for easier debug (#803)

fix the status for easier debug (#803)

* Bug: Limit Receipt Selection When Select By Root (#808)

* change 'where limit' on sql

* UnitTest-ApiService-GetBalances (#804)

* first commit (#807)

unit test on api handler `GetAccountBalance`

* #800 do mempool backup before executing derived query Rollback() method (#810)

* remove key from badger db after mempools backup restored (#811)

remove key from badger db after mempools backup restored (#811)

* add badger metrics to prometheus (#815)

* add badger metrics to prometheus

* explicitly use badger v2

* remove badgerdb v1

* revert MaxAPIRequestPerSecond to avoid nodes being overwhelmed by api calls

* enhance error logging for pushBlock and validateBlock process (#818)

* enhance error logging for pushBlock and validateBlock process

* exclude unimportant push and validate block error

* change block_height to reference_block_height (#820)

change block_height to reference_block_height (#820)

* Bug: Missing Rollback When BackupMempool Fail in PoppedOffBlock #802 (#816)

* first commit

* fix: Missing Rollback When BackupMempool Fail in PoppedOffBlock 

* change BackupMempols --> BackupMempools

* Unit test api handler account dataset (#821)

* GetAccountDatasets

* GetAccountDataset

* Expire musig pending transaction (#828)

* ignore .bak like genesis.go.bak :)

* expiring pending transactions by height and undo apply unconfirmed

* update readme about multis sig generator

* not allow filter by pending and expired

* finish query logic to, so now (block_height+min_rollback) = current_block_height

* defer func for rows close and matches the others about query logic

* add log instance into transaction core service, ignore log level

Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* Unit test api block handler (#830)

* TestBlockHandler_GetBlock

* TestBlockHandler_GetBlocks

* fix captLocal

* fix gocritic

* Unit test api mempool handler (#831)

* TestMempoolTransactionHandler_GetMempoolTransaction

* TestMempoolTransactionHandler_GetMempoolTransactions

* 829 download escrow transaction (#834)

* #832 add one more persistance time cycle to check to avoid node stuck in small network

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #833 add copy of block to broadcast before deleting the transactions

* #829 fix test

* #829 add test for escrow transaction query

* #833 broadcasted wrong variable

* #829 refactor escrow transaction query to be more generic

* Hotfix deepcopy broadcast block (#837)

* add mohae/deepcopy dependency

* enforce deep copy for broadcast block

* use map to reduce loop complexity (#838)

* get the latest account balances (#839)

Co-authored-by: iltoga <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Kevin Hadi <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
iltoga added a commit that referenced this pull request Jun 9, 2020
@astaphobia @iltoga
@andy-shi88 @sg2bcz @sukrawidhyawan @capt4ce @nawikart @KevinH2810
Escrow bug fixes, Musig and Genesis generator enhancements (#856)
f97f4b6 
* Code Refactoring (#587)

* Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths

* Get escrow transactions with filter fields and pagination (#590)

* get escrow transactions with filter fields and pagination

* get escrow transactions functional test

* #588 decouple published receipt and participation score from block main service (#591)

* 592 fee model (#595)

* #592 add fee model to action type

* #592 add fee model to action type

* #592 ignore package-interface naming lint rule

* #592 update pb file of transaction : add model + service for getTxMinimumFee api

* #592 add handler for calculate minimum fee api

* #592 fix GetTransactionMinimumFee request

* #592 fix wrong type

* #592 add api client for testing get minimum fee

* 578 snapshot file creation (#593)

* Logic for snapshot file creation

* Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain)
* Add ugorji/go/codec package to add CBOR serialization support

* Added Logic for snapshot file processing (create/download)

* Add new interface and service to abstract snapshot file download
* Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks

* Added business logic for snapshots

* Add new service FileService to abstract (and make testable) methods relative to file manipulation

* Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods)
* Move GetFileNameFromHash and GetHashFromFileName into fileservice
* Add query and service handler to retrieve account balances for snapshots
* Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP)
* Add initial logic to create a snapshot file containing all (only last state) account balances
* TODO: increase code coverage (few methods are tested so far)

* Bug fix

* Add EscrowTransactionQuery to GetDerivedQueries

* Added logic for generating snapshot files

* Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService
* Add test case for NewSnapshotFile and new queries

* Finalised logic for generating snapshot files (first iteration only has single file per snapshot)

* Add all relevant queries to get snapshot's payload
* Add method to parse and import snapshot into db
* Add config parameter for snapshots folder
* Add unit and integration tests

* Added logic for download snapshots

* Add method to parse the spineBlockManifest and download file chunk hashes
* TODO: download file is still mocked

* Uncomment logic previously commented for testing purposes
* Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments

* Add query interface for snapshot-specific queries (SnapshotQuery)
* Add SelectDataForSnapshot interface signature to new interface
* Add GetSnapshotQuery method to get all snapshot queries implementations (in a map)
* Refactor relative query methods in every relevant repository
* Refactor some constant to use time.Duration instead of int type
* Fix unit tests

* Add DESC to order by clauses in snapshot queries
* Refactor GetChainTypeCount to GetChainTypes to make chain types countable
* Remove temporary test function


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)

* 596 multisig skeleton (#597)

* #596 add multisignature transaction body

* #596 add constants for MultiSignature body fields

* #596 add multisignature tx type skeleton

* #596 add multisig tx type enum

* #596 add command tools for generate multisignature tx and parsing any tx

* #596 basic multisignature parsing functionality

* #596 update schema commit

* #596 add signature info

* #596 update signature structure to use map

* #596 adjust cmd tools to new multisig byte structure

* #596 update multisig transaction bytes

* #596 update schema

* 599 snapshot chunk strategy (#605)

* Added proto message for snapshots

* Add SnapshotPayload message representing internal snapshot data structure

* Added logic for snapshot chunk 'basic' strategy

* Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot
* Add SnapshotPayload model to schema, since we are using it in two different packages
* Refactor snapshotmainblockservice to use new snapshot chunk strategy
* Refactor unit and integration tests

* Completing logic for snapshot basic chunk strategy

* Finalize BuildSnapshotFromChunks and its test
* Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service

* Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile

* 600 multisig queries (#607)

* #600 add new managed table for multisig

* #600 update multisig managed table model

* #600 add managed table query for multisig

Co-authored-by: Ariasa Ketut <[email protected]>

* 602 generate multisig address (#609)

* method multisig account address assembler

* command line tool multisig generator command

* set 1.20.0 of golangci-lint version to fixed circleci fail

* no need to validate current block height (#610)

* inhibit spine blocks smithing process during main blocks first download (#608)

* Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started)
* Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService
* Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service
* Add new blocker type P2PNetworkConnectionErr
* Increase timers for checking block download process
* Refactor blockStatusService and implementations to blockTypeStatusService
* Refactor relative code in other services
* Remove cast to int snapshotPayloadHeight in snapshotMainBlockService
* Fix bug in snapshotPayloadHeight computation
* Refactor constructor method for BlockchainSyncService

* Fixed transaction generator affected by fee (#611)

* no need to validate current block height

* fixed transcation generator afftected by fee

* Code Refactoring and Bug Fixing (#613)

* Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded)
* Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status
* Remove unused getter from chainType
* Add new constants for better code readability
* Fix tests

* 601 multisig validation (#615)

* #601 add --post and --post-host to enable post transaction bytes on generated

* #601 add multisig validation

* #601 fix lint

* #601 wrong description in cmd

* Optimize GRPC duration (#616)

* no need to validate current block height

* set index into some tables for speed up latency get a lot of records

* remove unused a line of codes

* GRPC endpoint get single escrow transaction by ID (#620)

* no need to validate current block height

* functionality test get escrow by id

* Download snapshot chunks (#614)

* Change fatalf to errorf/return in goroutine in snapshotService.go
* Add missing return statement for error in snapshotService.go

* Added logic for downloading files from a random peer and Code Refactoring

* Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy
* Add rpc call (client request and server response) to download a file from a random peer
* Add logic to download/parse/verify a file and a snapshot

* Added logic and file download metrics

* Add logic to download a file, server side
* Add metrics to count succeeded/failed snapshot downloads

* Update schema: add Failed files for file download
* Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node
* Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones
* Rename megablockQuery to spineBlockManifestQuery
* Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex
* Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data
* Fix smith process lock when downloading spine and main blocks
* Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing


* Add retry download failed file chunks from different peers
* WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string

* Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them
* Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases)
* Optimize p2p.DownloadFilesFromPeer while writing the test
* Fix code after code review

* fixed bug in caclulating snapshot's height

* Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count)
* Refactor code that used sync.Map in favor of new map struct
* Add util function to calculate a quick (non secure) random number
* Fix uncaught nil in p2pServerHandler

* Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks)
* Add a couple of TODOs for spine block manifest validation

* Fix bug in unitialized model.MapIntBool struct
* Add missing test for GenerateSnapshot method

* Uncommented code for testing

* Uncomment code
* Fix one unit test after fixing the snapshot height (previous commit)

Co-authored-by: stefano galassi <[email protected]>

* 594 add bitcoin signature (#618)

* add bitcoin sig, btcd &  btcutil package

* restructure Ed25519 signature type function

* add cmd to generate bitcoin account

* add signature-type flag in cmd tx generator

* cmd enhace switch case generate account

* return error when sign and verify payload

* cmd generate tx with sender signature type

* adjust general parse transaction bytes

* adjust unit test

* udate cmd readme

* cmd, panic when invalid signature type

* remove unused function

* add generate account in signature

* add signature unit test

* add misisng import package

* update schema

* change parameter type of GenerateAccountFromSeed

Co-authored-by: Ariasa Ketut <[email protected]>

* 629 - Use gomod consistently (#630)

* no need to validate current block height

* update circleci config use go modules instead of dep and update go version to 1.14

* go mod and more cleaning shadow declaration variable

* Revert "GRPC endpoint get single escrow transaction by ID (#620)"

This reverts commit 7824924.

* Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)""

This reverts commit bedbf87.

* revert unused commit

* remove Gopkg.* files

* remove no needed comment

* make sure go modules cache correctly

* change circleci working directory, update readme

* update path of cache

* fixed go modules caches

* 603 multisig apply (#623)

* #603 initialize multisig service layer

* #603 implement apply confirmed

* #603 add sender address in pending transaction

* #603 update multisignature.PendingTransaction schema

* #603 delete multisig tx util

* #603 add sign command to support multisig development

* #603 add --sender-address flag for generating unsigned bytes

* #604 add sender address to unsigned tx

* #603 return raw error from execute transaction to enable sqlite error casting

* #603 complete multisig implementation

* #603 fix tests

* #603 add latest flag to pending transaction

* #603 update pending_transaction status on executed

* #603 add latest flag to pending transaction

* #603 add filter to pending transaction query

* #603 add latest flag to multisig-info and signatureInfo

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx model

* #603 add latest field to all pending tables

* #603 adjust query to handle latest flag

* #603 insert multisig-child transaction to transaction table

* #603 dfix test

* #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig)

* #603 fix example cmd

* #603 update schema

* #603 remove binary

* #603 convert address-Signature to format stringTostring

* #603 fix query inconsistency

* #603 handle unsigned transaction execution status

* #603 revert unnecessary change

* #603 handle submit multisig info execute multiple tx

* #603 fix query inconsistency

* #603 add multisig signature type

* #603 add constant for multisig signature validation type;

* #603 remove debug log

* CI delivery image version (#632)

* fixed invalid config

* change image version to stretch

* remove configuration for test purposes

* Ci delivery image version (#633)

* fixed typo

* 604 api rate limit (#625)

* add rate limiting feature for API

* fix unclosed rows

* change MaxAPIRequestPerSecond from int32 to uint32

* add download process metrics tracker

* add go sum

* 621 - Respecting  escrow expiration on Push Block (#626)

* no need to validate current block height

* fixing bug, respecting expiring escrow process on PushBlock

* fixed escrow rollback query and respecting escrow stage before push block process

* fixed escrow unit test result

* has been fixed forgot to close rows

* allow expiring escrow method to control db transaction

* Spine block manifest and spine public keys validation when downloading blocks from peers (#631)

* Bug fixing

* Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers
* Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash)
* Fix tests

* Code Refactoring

* Refactored method's name from GetPayloadHash to GetPayloadHashAndLength

* Added logic to validate block's payload on block push

* Add ValidatePayloadHash logic to blockMainService and blockSpineService
* Fix tests

* Add logic to validate spine block manifest against its relative spine block of reference

* Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp)
* Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network
* Add unit tests for new methods

* Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation

# Conflicts:
#	core/service/blockMainService.go

* Code Refactoring

* Update genesis block ID after changes to Generate block (added published receipts to payload length)
* Add unit test for validateblockhash

* Code Refactoring

* Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods

* Code Refactoring

* Removed unnecessary comment

* Bug fixing

* Remove fmt.println probably left from some internal test

* Bug fixing

* Update comment in query

* Code Optimization

* Optimize if statement by reversing conditions (courtesy of @ali)

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#636)

* Fixed bug in download snapshot process causing segfault

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#639)

* Fix bug in pendingSignatureQuery rollback function
* Remove unused function from transactionQuery

Co-authored-by: stefano galassi <[email protected]>

* Escrow query test fail fixed (#640)

* no need to validate current block height

* escrow query test fail fixed

* Bug Fixing (#642)

* Fix bug in unchecked map key leading to panic in block smith process

* Escrow query test fail fixed (#647)

* no need to validate current block height

* escrow query test fail fixed

* fixed escrow map params need to sorting before compare

* 634 multisig api (#645)

* #634 init multisig api

* #634 update multisig request model

* #634 update get pending tx detail response

* #634 update get pending tx detail response

* #634 initiate multisig client API

* #634 fix bug: wrong empty signature checks

* #634 remove built binary from repository

* #634 add multisig api tests

* #634 revert unnecessary changes

* #634 update go mod

* #634 initiate health check service and model

* #646 add health check endpoint

* #634 update schema

* #634 use getter instead of accessing field directly

* #634 add stricter param check

* #634 fix pagination

* #634 remove duplicate code

Co-authored-by: Ariasa Ketut <[email protected]>

* 634 multisig api (#653)



* #634 add default field to order field param

Co-authored-by: Ariasa Ketut <[email protected]>

* #651 fix multisig_address signature validation (#652)

* #651 fix multisig_address signature validation

* #651 fix test

Co-authored-by: Ariasa Ketut <[email protected]>

* 656 incorrect account ledger balance changes (#657)

* no need to validate current block height

* account ledger incorrect balance change information fixed

* 654 account ledger multisig (#658)

* #654 add multisig transaction event to account ledger

* #654 add ledger record in multisig transaction

* #654 update schema commit

Co-authored-by: Gede Sukra Widhyawan <[email protected]>

* 660 api get multisig info (#661)

* #660 add new api endpoint proto for getting multisignature info

* #660 implement get multisig info api

* #660 rename getPendingTransactions request

* #660 rename getPendingTransactions request

* #660 update schema commit

* 650 - CMD HD wallet generate account (#668)

* no need to validate current block height

* generate account slip10

* allow fetch from private repo

* fixed invalid format of ci

* fixed set env of machine

* add additional ssh key

* fixed bash condition when check gomod exists

* allowed github connection via token instead of ssh

* remove unused code

* more describe generate account command

* fix uninitialized metrics (#662)

fix uninitialized metrics (#662)

* bitcoin sig: add public key into signature (#670)

* bitcoin sig: add public key into signature

* handle multiple rows in a function (#675)

handle multiple rows in a function (#675)

* SelectReceipt Limit (#676)

SelectReceipt Limit (#676)

* Circleci GitHub token (#677)


* token as env variable

* 628 apply snapshot (#667)

* added alpha nodes genesis.go

* Rollback mempool - Restoring mempool transactions backup (#419)

* initiate restoreMempoolsBackup func

* finished mempool backup process and add loggger into processfork struct

* add logger into blockPopper stuff

* 420 get scramble nodes enhance (#423)

* #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now

* #420 move some scrambling function to util to reduce fetching function call frequency

* #420 add block_height to merkle tree table

* #420 fix query to fetch receipt, it had wrong block height scope on the query

* #420 handle error

* #420 add receipt validation

* #420 build scrambled node at height x

* #420 adjust constructor update

* #420 move receipt service instantiation to after node registration service

* #420 move scramble node model outside, it is used by other service as well : receipt service

* #420 validate receipt before storing to database

* #420 use bytes.equal for better comparison performance

* #420 rename memoized scramble nodes

* #420 simplify loop

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted broken query and kept changes from newer pr

* 432 revert participation score query (#437)

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted query

* reverted broken participation score query

* change a few configurations special for alpha (#434)

* change a few configurations special for alpha

* updated config.toml

* added (peer) port to genesis nodes addresses

* fixed test

* updated production values for some constants

* updated log levels for testing alpha nodes

* reset default log levels

* increased PriorityStrategyBuildScrambleNodesGap to 40

* fixed unwanted diffrences between develop and alpha branches

* fixed unit tests for alpha

* fixed unit test for mempoolCoreService

* Fixed unit test

* Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks

* switch log level for debugging purpose

* debug: process fork warn to errorf log

* revert: revert log levels

* keep track of p2p go routine metrics (#569)

* fix shadowed error message

* Resolved merge conflicts

* Code refactoring

* Updated mainchain genesis id

* Code Refactoring

* Update main block genesis ID

* Updated constant

* Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails

* Update golang.org/x/crypto/ed25519 to crypto/ed25519

* Reverted crypto/ed25519 to golang.org/x/crypto/ed25519

* Fixed bug in download snapshot process causing segfault

* Adding business logic

* Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo
* Add logic to snapshot service to include multisig data in snapshots


* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries

* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries
* Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported)
* Add logic to main.go to import snapshot after downloading it

* Refactor IsSnapshotHeight with simplified logic

* Refactor code to be aligned with develop branch instead of alpha

* Refactor SelectDataForSnapshot queries to use rollback-like logic

* Refactor SelectDataForSnapshot queries to use rollback-like logic
* Write unit tests for new queries

* Fix wrong table nale in new block query

* Fix wrong table nale in new block query

* Fix do not allow to delete genesis block when trimming block table

* Revert configX.toml log level to "fatal", "error", "panic"

* #628 add checks if all blocksmith time has expired

* #628 move push block to anonymous function

* Add rollback to snapshot height to set latest = 1 to all latest entitites

* Fixed SelectDataForSnapshot query in accountDataset repo

* Fix missing for loop break conditions in ScanBlockPool
* Add function to log running binary file checksum at node startup

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: stefano galassi <[email protected]>

* 672 makefile binary built (#678)

* #672 ignore release folder

* #672 initialize makefile

* #672 add github token example file

* #672 move xgo to zoobc repository

* #672 add makefile usage

* #672 add cc for cmd

* #672 add go-fmt and separate golangcilint version

* #672 move test to makefile command

* #672 fix gopath

* #672 update readme for github.token usage

* #672 print information for step ran

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* 665-Enhance CMD Database Init (#680)

* cmd: put init db only where is needed

* cmd: add commad to generate proof of ownership

* cmd: add readme rollback

* cmd: fixing remove node reference func

* handle nil value (#682)

* throw error on over limit get blocks (#684)

* throw error on over limit get blocks

* move limit validation to handler layer

* remove pending tx status change, it has been updated in apply confirmed (#685)

* 686: Cmd, enhance generate account (#687)

* cmd: enhance generate account

* add get public key using slip10

* 655 escrow receipt approval (#688)

* no need to validate current block height

* account dataset escrow approval and integration

* fullfill send money validation test cases

* to be gentle no need to check the account dataset property

* create specific method of query for account dataset

* getHostInfo unit testing (#689)

* getHostInfo unit testing
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* adding database status metrics (#693)

adding database status metrics (#693)

* Bug Fixing snapshot selection queries and enhancing rollback queries (#696)

* Bug Fixing

* Fix snapshot selection queries
* Enhance rollback queries
* Enhance db tables by adding composite primary keys where possible
* Fix bug in rollback and snapshot selection queries

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 681 disable smithing non registered node (#694)

* #681 add zero participation score blocker

* #681 inject blockchainStatusService to noderegistration service

* #681 add flag to say if a node is blocksmith or not

* #681 update main.go with new struct constructor

* #681 only update the blocksmith status when detect zero participation score

* #681 compare with bytes.equal()

* unit testing for GetBlockQueue (#698)

unit testing for GetBlockQueue (#698)

* Bug Fixing (#706)

* Fix breaking bug in rollback query
* Refactor spinePublicKey rollback query to comply to new code logic

* 691 grpc account dataset (#699)

* no need to validate current block height

* rename accountDataset to plural name

* account dataset grpc service list and single record

* grpc client example

* move params checkup to handler package

* remove unnecessary test cases

* technical error message passed

* 690 empty block smith (#703)

* #690 add empty flag to enable empty block generate

* #690 add limit for skipped block smith

* #690 set the skipped limit back to 15

* #690 remove unimplemented test

* #690 fix return arguments

* #690 remove unnecessary variable initialization

* #690 make use of default int64 value

* #690 smaller skipped blocksmith constant so we can see the effect on dev net

* adding function for HD wallet testing (#683)

* adding function for HD wallet testing

* P2P: Add unit test on server service (#707)

* p2p: add unit test p2p server service

* enhance Err format

* change GetPeerInfoResponse field name

* p2p: update unit test

* update schema

Co-authored-by: Andy Shi <[email protected]>

* added skippedblocksmith to snapshot (generate/apply) (#710)



* Add SkippedBlocksmith to snapshot proto model

* Added Logic for snapshots

* Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository

* Added Logic for snapshots

* Add logic to include skippedblocksmith records to snapshots
* Add logic to import skippedblocksmith records from snapshots into db (apply snapshot)
* Fix unit tests

* Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot

Co-authored-by: stefano galassi <[email protected]>

* Adding runtime cpu profiling for core node (#712)

* Adding Application Logic
* Add configuration parameter for profiling
* Add runtime flag to enable profiling
* Add logic to run cpu profiling
* Update configuration parameter for profiling
* Add logic to enable memory profiling
* Update documentation (readme.md): see 'run' paragraph

* add more info in log (#713)

* fixed inconsistance block height on escrowed transaction (#717)

* Code Refactoring (#716)

* Use net/http/pprof package that serves the full profiling api via http
* Enable profiling api only when running app with -cpu-profile flag (see updated readme)
* Refactor the way monitoring server is instantiated: not using default http hanlder anymore
* Update readme.md

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#719)

* Refactor block query to select a range of blocsk instad of a single one for snapshots
* Refactor service logic to add more blocks (720) to snapshots

* Cleaning redundant function (#714)

* adding function for HD wallet testing

* cleaning some unused variables

* remove unused const and redundant function on generator

* Bug Fixing (#724)

* Fix pprof http server listening only on localhost

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#728)

* Optimize function GenerateTxRemoveNode

Co-authored-by: stefano galassi <[email protected]>

* 722 blockchain orchestrator (#726)

* add BlockchainOrchestrator to control the chains

* 709 Smithing All Blocksmith Skipped (#725)

* #709 add smithing pending blocker

* #709 remove smith time from blcoksmith model

* #709 update smithing process: recycle blocksmith when all blocksmith time are passed

* #709 separate smith time constants between spinechain and mainchain

* #709 inject spinechain blocksmith service

* #709 update metric smithtime to smith index

* #709 cover new smithing process unit tests

* #709 fix comment on blocksmith strategy & update fake smith function signature

* add index on block table and update sql connnection number (#729)

add index on block table and update sql connnection number (#729)

* scale down the common block jump (#731)

scale down the common block jump (#731)

* 721 - Remove Account Dataset Expiration (#732)

* accountDatasetsQuery should a plural name

* update affected services and more about account_dataset stuff

* fixing unit test of NewSnapshotFile integration

* escrow validation timeout limit exceeded

* Add node registry partial history to snapshots (#734)

* Code Refactoring

* Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process
* Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter
* Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve:
  * latest state from block 0 to 'fromHeight'
  * full history fromHeight to toHeight
  * merge and remove duplicate ids in first selection
* Remove unused constant LinkedReceiptBlocksLimit

Co-authored-by: stefano galassi <[email protected]>

* making the download process less restrictive to failures in catching  up with peers (#735)

* making the download process less restrictive to failures in catching up with peers

* fix the way of tracking errors

* fix the way of keeping track the error

* TestBlockIncompleteQueueService_AddTransaction (#720)

* TestBlockIncompleteQueueService_AddTransaction

* add succes case

* fix return type to use blocker (#742)

* fix return type to use blocker

* add more blocker type

* Update golangci to 1.24.0 (#743)

* update golangci to 1.24

* add reset db variable step

* fixed invalid space

* test circlleci building a branch

* fixes golangci warn and error

* disable ineffassign

* adjust the deployment step

* #740 index receipt fields for better query time (#744)

* #740 index receipt fields for faster query

* #740 verbose message on golangcilint run

* fix error casting (#747)

* fix error casting

* revert p2p error type

* fix blocker usage

* downgrade golangci-lint version (#748)

* Hotfix rows immediate release (#749)

* immediately release rows after converted to model

* add unique index to published_receipt.datum_hash

* fix: wrong column name

* temporary fix, omit the unique

* Unit Testing core service BlockPoolService (#739)

* NewBlockPoolService, GetBlocks, GetBlock, InsertBlock, ClearBlockPool

* add assertion

* gofmt -s -w

* Add optional Params on Sign function (#746)

* enhance sign function & cmd signature

* update readme

* add option params on GenerateSignedTxBytes

* update comment function SignEd25519

* add return err on parseBytesArgs func

* fix type & unused arg

* missing flag ed25519UseSlip10

* handle error from block push and validation (#751)

handle error from block push and validation (#751)

* fix block validation and smith time calculation (#752)

* setter and recipient must be the same person (#755)

* Fix: P2P - Can't Add UnresolvedPeers  (#757)

* add condition when there are still space in unresolved peers

* remove duplicate removal query (#758)

* Optimize database transaction (#756)

* use ExecuteSelectRow instead

* use QueryRow instead of Query, get single account balance

* use QueryRow instead if only want a single record

* nodeReg.ApplyConfirmed better to use QueryRow

* nodeRegUpdate.ApplyConfirmed better to use QueryRow

* removeNodeReg.ApplyConfirmed better to use QueryRow

* CMD fix and addition (#762)

* add blockheight logging at the end of applying a snapshot

* fix rollback cmd method to use the parameter passed

* add cmd methods for getting scrambledNodes and priorityPeers

* rename folder of scrambled nodes

* Use QueryRow instead of Query when getting single record (#764)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix query and the unit tests (#765)

fix query and the unit tests (#765)

* Confirm the state of download finish if the whole network stuck (#770)

* confirm the state of download finish if the whole network stuck

* reset the firstDownloadCounter when finding out they are not stuck

* reduce the validation limit constant

* Unit Test Transaction Core Service (#767)

* UndoApplyUnconfirmedTransaction

* ApplyConfirmedTransaction

* ApplyUnconfirmedTransaction

* ValidateTransaction

* remove underscores in Go names

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* GetBlockByHeight: Query row instead of Query (#768)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

* GetBlockByHeight better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix finish download (#771)

fix finish download (#771)

* reset db automatically and also snapshots stuff (#772)

* reset db automatically and also snapshots stuff

* *_kv would remove also

* snapshots suffix

* Fix manifest rollback (#773)

* add spineblock height field

* rename spine block manifest column and add indesx on block height column

* rename spine_block_manifest column field

* move spine_block_manifest to spinechain derived query category

* clean snapshot file on rollbacked

* add delete snapshot file function

* update tests

* remove ErrNoRows check as that error only occur on sql.Row not sql.Rows

* inject snapshot block service to spine block service

* snapshot manifest field rename

* fix new rollback query for manifest

* use parse instead of splitting chunks manually

* log the deletion error

* handle error

* update schema

* add extra comment for testing

* move file deletion to after db commit and as goroutine

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* #777 update validate block function signature : remove timestamp (#779)

* #777 update validate block function signature : remove timestamp

* move the validation of download finish if the block height is 0 (#781)

move the validation of download finish if the block height is 0 (#781)

* Bug limit receipt selection when select by root 774 (#776)

* versi 1

* small edit

* update mock ExecuteSelect

* make sure the query is similar as mock

* make it simpler

* merge schema

Co-authored-by: Nawi Kartini <[email protected]>

* 782 missing blocks fix (#783)

* #782 validate block in scanBlockPool thread

* #782 fix wrong function signature causing interface not implemented

* #782 move chain lock to beginning of ScanBlockPool function

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* Getting own public ip (#784)

* get own public ip

* allowing to get public ip via checkip.dyndns.org

* make sure the parsed ip is valid ip address

* IPUtil methods

* Fixing error nil pointer when accessing sql.Tx Executor (#786)

* add missing err checker on validate mempool

* fixing racing unlocking mutex & making sql.Tx nil

* Sort poppedBlocks by height ascending (#787)

* sort poppedBlocks by height desc

* fulfill test cases of PoppedoffBlocks

* sort ascending popped off blocks

* fulfill spine PopOffToBlock test cases

* #792 replace spendable with balance (#793)

* Genesis generate command: Code Refactoring and adding new feature to facilitate deployments with nomad/consul (#797)

* Code Refactoring and adding new feature

* Refactor command code to comply to new standards (see other commands)
* Add feature to generate consulKvInit.sh script from a template and a the newly generated genesis

* Bug fixing

* Limiter must be 2*MinRollbackHeight (#795)

* limiter must be 2*MinRollbackHeight

* 789 multisig pending transaction Apply-Flow (#798)

* #789 update pending_transaction query by hash to have multiple status filter

* #789 update flow of pending_transaction execution

* Tech: GetAccountBalances #785 (#788)

* first commit

* check num of addresses

* fix

* fix

* fix

* fix the unit testing

* use caseQuery instead of looping

* fix caseQuery.In

* fix unit testing

* move validation to handler

* fix unit test part 1

* auto update from proto

* accountBalance

* change to status.Error

* change to plural

* refine according PR comment

* fix

* fix unit test part 1

* remove commented line

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* fix the status for easier debug (#803)

fix the status for easier debug (#803)

* Bug: Limit Receipt Selection When Select By Root (#808)

* change 'where limit' on sql

* UnitTest-ApiService-GetBalances (#804)

* first commit (#807)

unit test on api handler `GetAccountBalance`

* #800 do mempool backup before executing derived query Rollback() method (#810)

* remove key from badger db after mempools backup restored (#811)

remove key from badger db after mempools backup restored (#811)

* add badger metrics to prometheus (#815)

* add badger metrics to prometheus

* explicitly use badger v2

* remove badgerdb v1

* revert MaxAPIRequestPerSecond to avoid nodes being overwhelmed by api calls

* enhance error logging for pushBlock and validateBlock process (#818)

* enhance error logging for pushBlock and validateBlock process

* exclude unimportant push and validate block error

* change block_height to reference_block_height (#820)

change block_height to reference_block_height (#820)

* Bug: Missing Rollback When BackupMempool Fail in PoppedOffBlock #802 (#816)

* first commit

* fix: Missing Rollback When BackupMempool Fail in PoppedOffBlock 

* change BackupMempols --> BackupMempools

* Unit test api handler account dataset (#821)

* GetAccountDatasets

* GetAccountDataset

* Expire musig pending transaction (#828)

* ignore .bak like genesis.go.bak :)

* expiring pending transactions by height and undo apply unconfirmed

* update readme about multis sig generator

* not allow filter by pending and expired

* finish query logic to, so now (block_height+min_rollback) = current_block_height

* defer func for rows close and matches the others about query logic

* add log instance into transaction core service, ignore log level

Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* Unit test api block handler (#830)

* TestBlockHandler_GetBlock

* TestBlockHandler_GetBlocks

* fix captLocal

* fix gocritic

* Unit test api mempool handler (#831)

* TestMempoolTransactionHandler_GetMempoolTransaction

* TestMempoolTransactionHandler_GetMempoolTransactions

* 829 download escrow transaction (#834)

* #832 add one more persistance time cycle to check to avoid node stuck in small network

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #833 add copy of block to broadcast before deleting the transactions

* #829 fix test

* #829 add test for escrow transaction query

* #833 broadcasted wrong variable

* #829 refactor escrow transaction query to be more generic

* Hotfix deepcopy broadcast block (#837)

* add mohae/deepcopy dependency

* enforce deep copy for broadcast block

* use map to reduce loop complexity (#838)

* get the latest account balances (#839)

* Unit test api node admin handler (#840)

* TestNodeAdminHandler_GetProofOfOwnership

* TestNodeAdminHandler_GenerateNodeKey

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 846 escrow inconsistent (#847)

* #846 add query to fetch expired escrow transaction at given height

* #846 save escrow_transaction.timeout as it is (submitted by client / node)

* #846 modify expiring escrow transaction function to use the new query

Co-authored-by: Ariasa Ketut <[email protected]>

* RequestBlockTransactions must include escrow part (#848)

* now requesting block transactions will include escrows

* inconsistence indexing of map test cases fixes

* might be only pending ecrow tx included

* Bug: Error RESOURCE_EXHAUSTED GetTransactions (#849)

* Pagination can't be empty

* set default pagination

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* Unit test api multisig handler (#841)

* TestMultisigHandler_GetPendingTransactions

* TestMultisigHandler_GetPendingTransactionDetailByTransactionHash

* TestMultisigHandler_GetMultisignatureInfo

* fix error ci/circleci

* fix error ci/circleci

* remove unuse print

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* Unit Test Api TransactionHandler (#850)

* first commit

* fix misspelling and too long line of codes

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* first commit (#851)

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* Re-ApplyUnconfirmed pending transactions (#853)

* re apply unconfirmed pending transactions after store snapshot manifest to db

* use log instead of fmt.Printf and remove unused var

* set current height to pending transaction that would be do apply unconfirmed

* 844 multisig changes (#854)

* #844 separate multisignature check to helper

* #844 refactor multisig
- update multisig rules
- move complicated database access to helper function

* #844 more test cases for multisig

* return array of pending signatures and pending transactions properly

* fix: checkMultisigComplete consider offchain multisig

* #844 index some multisig tables

* #844 test for helpers

* fix order of chain statuses in the returned response (#862)

fix order of chain statuses in the returned response (#862)

* Unit Test P2p handler p2pServerHandler.go (#863)

* GetMorePeers

* SendPeers

* GetCommonMilestoneBlockID

* GetNextBlocks

* GetPeerInfo

* GetCumulativeDifficulty

* GetNextBlockIDs

* SendBlock

* SendTransaction

* SendBlockTransactions

* RequestBlockTransactions

* Fix executor usages (#870)

* fix: handle duplicate rollback : level(panic)

* fix: handle rollback+commit() call in cmd.rollbackBlockchain : level(panic)

* fix: handle missing Executor.Rollback() call  : level(deadlock)

* fix: handle missing Executor.Rollback() call  : level(deadlock)

* fix: handle Commit() after Rollback()  : level(panic)

* CMD genesis generator and update circleci config (#865)

* cmd genesis generator and update circleci config

* fix golangci warning

* remove unused constant

* build job only for staging and develop

* account balance 0 as default

* wrong path given when get root via wd

* Dynamic genesis (#873)

* fixing typo CIRCLECI_BRANCH

* first commit (#866)

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* Unit test p2p native strategy (#868)

* GetMorePeersHandler

* GetResolvedPeers

* GetAnyResolvedPeer

* AddToResolvedPeer

* RemoveResolvedPeer

* GetUnresolvedPeers

* AddToUnresolvedPeer

* AddToUnresolvedPeers

* RemoveUnresolvedPeer

* GetBlacklistedPeers

* AddToBlacklistPeer

* RemoveBlacklistedPeer

* GetAnyKnownPeer

* PeerBlacklist

* PeerUnblacklist

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

Co-authored-by: iltoga <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Kevin Hadi <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@iltoga iltoga iltoga approved these changes

@andy-shi88 andy-shi88 andy-shi88 approved these changes

@sukrawidhyawan sukrawidhyawan sukrawidhyawan approved these changes

@astaphobia astaphobia Awaiting requested review from astaphobia

Assignees

@capt4ce capt4ce

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@capt4ce @iltoga @andy-shi88 @sukrawidhyawan