Skip to content

Conversation

capt4ce
Copy link
Contributor

@capt4ce capt4ce commented May 5, 2020

Description

Differentiate processes in the processing received block to ease debugging stuck node

Breakdown

  • Change position of metrics logging in Unlock in the blockService
  • Add additional blockchain status BlockchainStatusReceivingBlockScanBlockPool and BlockchainStatusReceivingBlockProcessCompletedBlock

Reference Issue

Close #numberIssue

Step to Test (optional)

  • step 1
  • step 2
  • step 3

@capt4ce capt4ce added ENHANCE HIGH_PRIORITY needed ASAP for current sprint labels May 5, 2020
@capt4ce capt4ce self-assigned this May 5, 2020
Copy link
Contributor

@iltoga iltoga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@andy-shi88 andy-shi88 merged commit b476890 into develop May 6, 2020
@andy-shi88 andy-shi88 deleted the differentiate-received-block-status-metrics branch May 6, 2020 08:33
andy-shi88 added a commit that referenced this pull request May 14, 2020
* Code Refactoring (#587)

* Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths

* Get escrow transactions with filter fields and pagination (#590)

* get escrow transactions with filter fields and pagination

* get escrow transactions functional test

* #588 decouple published receipt and participation score from block main service (#591)

* 592 fee model (#595)

* #592 add fee model to action type

* #592 add fee model to action type

* #592 ignore package-interface naming lint rule

* #592 update pb file of transaction : add model + service for getTxMinimumFee api

* #592 add handler for calculate minimum fee api

* #592 fix GetTransactionMinimumFee request

* #592 fix wrong type

* #592 add api client for testing get minimum fee

* 578 snapshot file creation (#593)

* Logic for snapshot file creation

* Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain)
* Add ugorji/go/codec package to add CBOR serialization support

* Added Logic for snapshot file processing (create/download)

* Add new interface and service to abstract snapshot file download
* Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks

* Added business logic for snapshots

* Add new service FileService to abstract (and make testable) methods relative to file manipulation

* Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods)
* Move GetFileNameFromHash and GetHashFromFileName into fileservice
* Add query and service handler to retrieve account balances for snapshots
* Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP)
* Add initial logic to create a snapshot file containing all (only last state) account balances
* TODO: increase code coverage (few methods are tested so far)

* Bug fix

* Add EscrowTransactionQuery to GetDerivedQueries

* Added logic for generating snapshot files

* Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService
* Add test case for NewSnapshotFile and new queries

* Finalised logic for generating snapshot files (first iteration only has single file per snapshot)

* Add all relevant queries to get snapshot's payload
* Add method to parse and import snapshot into db
* Add config parameter for snapshots folder
* Add unit and integration tests

* Added logic for download snapshots

* Add method to parse the spineBlockManifest and download file chunk hashes
* TODO: download file is still mocked

* Uncomment logic previously commented for testing purposes
* Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments

* Add query interface for snapshot-specific queries (SnapshotQuery)
* Add SelectDataForSnapshot interface signature to new interface
* Add GetSnapshotQuery method to get all snapshot queries implementations (in a map)
* Refactor relative query methods in every relevant repository
* Refactor some constant to use time.Duration instead of int type
* Fix unit tests

* Add DESC to order by clauses in snapshot queries
* Refactor GetChainTypeCount to GetChainTypes to make chain types countable
* Remove temporary test function


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)

* 596 multisig skeleton (#597)

* #596 add multisignature transaction body

* #596 add constants for MultiSignature body fields

* #596 add multisignature tx type skeleton

* #596 add multisig tx type enum

* #596 add command tools for generate multisignature tx and parsing any tx

* #596 basic multisignature parsing functionality

* #596 update schema commit

* #596 add signature info

* #596 update signature structure to use map

* #596 adjust cmd tools to new multisig byte structure

* #596 update multisig transaction bytes

* #596 update schema

* 599 snapshot chunk strategy (#605)

* Added proto message for snapshots

* Add SnapshotPayload message representing internal snapshot data structure

* Added logic for snapshot chunk 'basic' strategy

* Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot
* Add SnapshotPayload model to schema, since we are using it in two different packages
* Refactor snapshotmainblockservice to use new snapshot chunk strategy
* Refactor unit and integration tests

* Completing logic for snapshot basic chunk strategy

* Finalize BuildSnapshotFromChunks and its test
* Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service

* Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile

* 600 multisig queries (#607)

* #600 add new managed table for multisig

* #600 update multisig managed table model

* #600 add managed table query for multisig

Co-authored-by: Ariasa Ketut <[email protected]>

* 602 generate multisig address (#609)

* method multisig account address assembler

* command line tool multisig generator command

* set 1.20.0 of golangci-lint version to fixed circleci fail

* no need to validate current block height (#610)

* inhibit spine blocks smithing process during main blocks first download (#608)

* Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started)
* Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService
* Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service
* Add new blocker type P2PNetworkConnectionErr
* Increase timers for checking block download process
* Refactor blockStatusService and implementations to blockTypeStatusService
* Refactor relative code in other services
* Remove cast to int snapshotPayloadHeight in snapshotMainBlockService
* Fix bug in snapshotPayloadHeight computation
* Refactor constructor method for BlockchainSyncService

* Fixed transaction generator affected by fee (#611)

* no need to validate current block height

* fixed transcation generator afftected by fee

* Code Refactoring and Bug Fixing (#613)

* Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded)
* Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status
* Remove unused getter from chainType
* Add new constants for better code readability
* Fix tests

* 601 multisig validation (#615)

* #601 add --post and --post-host to enable post transaction bytes on generated

* #601 add multisig validation

* #601 fix lint

* #601 wrong description in cmd

* Optimize GRPC duration (#616)

* no need to validate current block height

* set index into some tables for speed up latency get a lot of records

* remove unused a line of codes

* GRPC endpoint get single escrow transaction by ID (#620)

* no need to validate current block height

* functionality test get escrow by id

* Download snapshot chunks (#614)

* Change fatalf to errorf/return in goroutine in snapshotService.go
* Add missing return statement for error in snapshotService.go

* Added logic for downloading files from a random peer and Code Refactoring

* Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy
* Add rpc call (client request and server response) to download a file from a random peer
* Add logic to download/parse/verify a file and a snapshot

* Added logic and file download metrics

* Add logic to download a file, server side
* Add metrics to count succeeded/failed snapshot downloads

* Update schema: add Failed files for file download
* Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node
* Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones
* Rename megablockQuery to spineBlockManifestQuery
* Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex
* Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data
* Fix smith process lock when downloading spine and main blocks
* Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing


* Add retry download failed file chunks from different peers
* WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string

* Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them
* Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases)
* Optimize p2p.DownloadFilesFromPeer while writing the test
* Fix code after code review

* fixed bug in caclulating snapshot's height

* Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count)
* Refactor code that used sync.Map in favor of new map struct
* Add util function to calculate a quick (non secure) random number
* Fix uncaught nil in p2pServerHandler

* Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks)
* Add a couple of TODOs for spine block manifest validation

* Fix bug in unitialized model.MapIntBool struct
* Add missing test for GenerateSnapshot method

* Uncommented code for testing

* Uncomment code
* Fix one unit test after fixing the snapshot height (previous commit)

Co-authored-by: stefano galassi <[email protected]>

* 594 add bitcoin signature (#618)

* add bitcoin sig, btcd &  btcutil package

* restructure Ed25519 signature type function

* add cmd to generate bitcoin account

* add signature-type flag in cmd tx generator

* cmd enhace switch case generate account

* return error when sign and verify payload

* cmd generate tx with sender signature type

* adjust general parse transaction bytes

* adjust unit test

* udate cmd readme

* cmd, panic when invalid signature type

* remove unused function

* add generate account in signature

* add signature unit test

* add misisng import package

* update schema

* change parameter type of GenerateAccountFromSeed

Co-authored-by: Ariasa Ketut <[email protected]>

* 629 - Use gomod consistently (#630)

* no need to validate current block height

* update circleci config use go modules instead of dep and update go version to 1.14

* go mod and more cleaning shadow declaration variable

* Revert "GRPC endpoint get single escrow transaction by ID (#620)"

This reverts commit 7824924.

* Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)""

This reverts commit bedbf87.

* revert unused commit

* remove Gopkg.* files

* remove no needed comment

* make sure go modules cache correctly

* change circleci working directory, update readme

* update path of cache

* fixed go modules caches

* 603 multisig apply (#623)

* #603 initialize multisig service layer

* #603 implement apply confirmed

* #603 add sender address in pending transaction

* #603 update multisignature.PendingTransaction schema

* #603 delete multisig tx util

* #603 add sign command to support multisig development

* #603 add --sender-address flag for generating unsigned bytes

* #604 add sender address to unsigned tx

* #603 return raw error from execute transaction to enable sqlite error casting

* #603 complete multisig implementation

* #603 fix tests

* #603 add latest flag to pending transaction

* #603 update pending_transaction status on executed

* #603 add latest flag to pending transaction

* #603 add filter to pending transaction query

* #603 add latest flag to multisig-info and signatureInfo

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx model

* #603 add latest field to all pending tables

* #603 adjust query to handle latest flag

* #603 insert multisig-child transaction to transaction table

* #603 dfix test

* #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig)

* #603 fix example cmd

* #603 update schema

* #603 remove binary

* #603 convert address-Signature to format stringTostring

* #603 fix query inconsistency

* #603 handle unsigned transaction execution status

* #603 revert unnecessary change

* #603 handle submit multisig info execute multiple tx

* #603 fix query inconsistency

* #603 add multisig signature type

* #603 add constant for multisig signature validation type;

* #603 remove debug log

* CI delivery image version (#632)

* fixed invalid config

* change image version to stretch

* remove configuration for test purposes

* Ci delivery image version (#633)

* fixed typo

* 604 api rate limit (#625)

* add rate limiting feature for API

* fix unclosed rows

* change MaxAPIRequestPerSecond from int32 to uint32

* add download process metrics tracker

* add go sum

* 621 - Respecting  escrow expiration on Push Block (#626)

* no need to validate current block height

* fixing bug, respecting expiring escrow process on PushBlock

* fixed escrow rollback query and respecting escrow stage before push block process

* fixed escrow unit test result

* has been fixed forgot to close rows

* allow expiring escrow method to control db transaction

* Spine block manifest and spine public keys validation when downloading blocks from peers (#631)

* Bug fixing

* Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers
* Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash)
* Fix tests

* Code Refactoring

* Refactored method's name from GetPayloadHash to GetPayloadHashAndLength

* Added logic to validate block's payload on block push

* Add ValidatePayloadHash logic to blockMainService and blockSpineService
* Fix tests

* Add logic to validate spine block manifest against its relative spine block of reference

* Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp)
* Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network
* Add unit tests for new methods

* Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation

# Conflicts:
#	core/service/blockMainService.go

* Code Refactoring

* Update genesis block ID after changes to Generate block (added published receipts to payload length)
* Add unit test for validateblockhash

* Code Refactoring

* Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods

* Code Refactoring

* Removed unnecessary comment

* Bug fixing

* Remove fmt.println probably left from some internal test

* Bug fixing

* Update comment in query

* Code Optimization

* Optimize if statement by reversing conditions (courtesy of @ali)

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#636)

* Fixed bug in download snapshot process causing segfault

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#639)

* Fix bug in pendingSignatureQuery rollback function
* Remove unused function from transactionQuery

Co-authored-by: stefano galassi <[email protected]>

* Escrow query test fail fixed (#640)

* no need to validate current block height

* escrow query test fail fixed

* Bug Fixing (#642)

* Fix bug in unchecked map key leading to panic in block smith process

* Escrow query test fail fixed (#647)

* no need to validate current block height

* escrow query test fail fixed

* fixed escrow map params need to sorting before compare

* 634 multisig api (#645)

* #634 init multisig api

* #634 update multisig request model

* #634 update get pending tx detail response

* #634 update get pending tx detail response

* #634 initiate multisig client API

* #634 fix bug: wrong empty signature checks

* #634 remove built binary from repository

* #634 add multisig api tests

* #634 revert unnecessary changes

* #634 update go mod

* #634 initiate health check service and model

* #646 add health check endpoint

* #634 update schema

* #634 use getter instead of accessing field directly

* #634 add stricter param check

* #634 fix pagination

* #634 remove duplicate code

Co-authored-by: Ariasa Ketut <[email protected]>

* 634 multisig api (#653)



* #634 add default field to order field param

Co-authored-by: Ariasa Ketut <[email protected]>

* #651 fix multisig_address signature validation (#652)

* #651 fix multisig_address signature validation

* #651 fix test

Co-authored-by: Ariasa Ketut <[email protected]>

* 656 incorrect account ledger balance changes (#657)

* no need to validate current block height

* account ledger incorrect balance change information fixed

* 654 account ledger multisig (#658)

* #654 add multisig transaction event to account ledger

* #654 add ledger record in multisig transaction

* #654 update schema commit

Co-authored-by: Gede Sukra Widhyawan <[email protected]>

* 660 api get multisig info (#661)

* #660 add new api endpoint proto for getting multisignature info

* #660 implement get multisig info api

* #660 rename getPendingTransactions request

* #660 rename getPendingTransactions request

* #660 update schema commit

* 650 - CMD HD wallet generate account (#668)

* no need to validate current block height

* generate account slip10

* allow fetch from private repo

* fixed invalid format of ci

* fixed set env of machine

* add additional ssh key

* fixed bash condition when check gomod exists

* allowed github connection via token instead of ssh

* remove unused code

* more describe generate account command

* fix uninitialized metrics (#662)

fix uninitialized metrics (#662)

* bitcoin sig: add public key into signature (#670)

* bitcoin sig: add public key into signature

* handle multiple rows in a function (#675)

handle multiple rows in a function (#675)

* SelectReceipt Limit (#676)

SelectReceipt Limit (#676)

* Circleci GitHub token (#677)


* token as env variable

* 628 apply snapshot (#667)

* added alpha nodes genesis.go

* Rollback mempool - Restoring mempool transactions backup (#419)

* initiate restoreMempoolsBackup func

* finished mempool backup process and add loggger into processfork struct

* add logger into blockPopper stuff

* 420 get scramble nodes enhance (#423)

* #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now

* #420 move some scrambling function to util to reduce fetching function call frequency

* #420 add block_height to merkle tree table

* #420 fix query to fetch receipt, it had wrong block height scope on the query

* #420 handle error

* #420 add receipt validation

* #420 build scrambled node at height x

* #420 adjust constructor update

* #420 move receipt service instantiation to after node registration service

* #420 move scramble node model outside, it is used by other service as well : receipt service

* #420 validate receipt before storing to database

* #420 use bytes.equal for better comparison performance

* #420 rename memoized scramble nodes

* #420 simplify loop

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted broken query and kept changes from newer pr

* 432 revert participation score query (#437)

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted query

* reverted broken participation score query

* change a few configurations special for alpha (#434)

* change a few configurations special for alpha

* updated config.toml

* added (peer) port to genesis nodes addresses

* fixed test

* updated production values for some constants

* updated log levels for testing alpha nodes

* reset default log levels

* increased PriorityStrategyBuildScrambleNodesGap to 40

* fixed unwanted diffrences between develop and alpha branches

* fixed unit tests for alpha

* fixed unit test for mempoolCoreService

* Fixed unit test

* Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks

* switch log level for debugging purpose

* debug: process fork warn to errorf log

* revert: revert log levels

* keep track of p2p go routine metrics (#569)

* fix shadowed error message

* Resolved merge conflicts

* Code refactoring

* Updated mainchain genesis id

* Code Refactoring

* Update main block genesis ID

* Updated constant

* Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails

* Update golang.org/x/crypto/ed25519 to crypto/ed25519

* Reverted crypto/ed25519 to golang.org/x/crypto/ed25519

* Fixed bug in download snapshot process causing segfault

* Adding business logic

* Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo
* Add logic to snapshot service to include multisig data in snapshots


* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries

* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries
* Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported)
* Add logic to main.go to import snapshot after downloading it

* Refactor IsSnapshotHeight with simplified logic

* Refactor code to be aligned with develop branch instead of alpha

* Refactor SelectDataForSnapshot queries to use rollback-like logic

* Refactor SelectDataForSnapshot queries to use rollback-like logic
* Write unit tests for new queries

* Fix wrong table nale in new block query

* Fix wrong table nale in new block query

* Fix do not allow to delete genesis block when trimming block table

* Revert configX.toml log level to "fatal", "error", "panic"

* #628 add checks if all blocksmith time has expired

* #628 move push block to anonymous function

* Add rollback to snapshot height to set latest = 1 to all latest entitites

* Fixed SelectDataForSnapshot query in accountDataset repo

* Fix missing for loop break conditions in ScanBlockPool
* Add function to log running binary file checksum at node startup

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: stefano galassi <[email protected]>

* 672 makefile binary built (#678)

* #672 ignore release folder

* #672 initialize makefile

* #672 add github token example file

* #672 move xgo to zoobc repository

* #672 add makefile usage

* #672 add cc for cmd

* #672 add go-fmt and separate golangcilint version

* #672 move test to makefile command

* #672 fix gopath

* #672 update readme for github.token usage

* #672 print information for step ran

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* 665-Enhance CMD Database Init (#680)

* cmd: put init db only where is needed

* cmd: add commad to generate proof of ownership

* cmd: add readme rollback

* cmd: fixing remove node reference func

* handle nil value (#682)

* throw error on over limit get blocks (#684)

* throw error on over limit get blocks

* move limit validation to handler layer

* remove pending tx status change, it has been updated in apply confirmed (#685)

* 686: Cmd, enhance generate account (#687)

* cmd: enhance generate account

* add get public key using slip10

* 655 escrow receipt approval (#688)

* no need to validate current block height

* account dataset escrow approval and integration

* fullfill send money validation test cases

* to be gentle no need to check the account dataset property

* create specific method of query for account dataset

* getHostInfo unit testing (#689)

* getHostInfo unit testing
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* adding database status metrics (#693)

adding database status metrics (#693)

* Bug Fixing snapshot selection queries and enhancing rollback queries (#696)

* Bug Fixing

* Fix snapshot selection queries
* Enhance rollback queries
* Enhance db tables by adding composite primary keys where possible
* Fix bug in rollback and snapshot selection queries

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 681 disable smithing non registered node (#694)

* #681 add zero participation score blocker

* #681 inject blockchainStatusService to noderegistration service

* #681 add flag to say if a node is blocksmith or not

* #681 update main.go with new struct constructor

* #681 only update the blocksmith status when detect zero participation score

* #681 compare with bytes.equal()

* unit testing for GetBlockQueue (#698)

unit testing for GetBlockQueue (#698)

* Bug Fixing (#706)

* Fix breaking bug in rollback query
* Refactor spinePublicKey rollback query to comply to new code logic

* 691 grpc account dataset (#699)

* no need to validate current block height

* rename accountDataset to plural name

* account dataset grpc service list and single record

* grpc client example

* move params checkup to handler package

* remove unnecessary test cases

* technical error message passed

* 690 empty block smith (#703)

* #690 add empty flag to enable empty block generate

* #690 add limit for skipped block smith

* #690 set the skipped limit back to 15

* #690 remove unimplemented test

* #690 fix return arguments

* #690 remove unnecessary variable initialization

* #690 make use of default int64 value

* #690 smaller skipped blocksmith constant so we can see the effect on dev net

* adding function for HD wallet testing (#683)

* adding function for HD wallet testing

* P2P: Add unit test on server service (#707)

* p2p: add unit test p2p server service

* enhance Err format

* change GetPeerInfoResponse field name

* p2p: update unit test

* update schema

Co-authored-by: Andy Shi <[email protected]>

* added skippedblocksmith to snapshot (generate/apply) (#710)



* Add SkippedBlocksmith to snapshot proto model

* Added Logic for snapshots

* Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository

* Added Logic for snapshots

* Add logic to include skippedblocksmith records to snapshots
* Add logic to import skippedblocksmith records from snapshots into db (apply snapshot)
* Fix unit tests

* Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot

Co-authored-by: stefano galassi <[email protected]>

* Adding runtime cpu profiling for core node (#712)

* Adding Application Logic
* Add configuration parameter for profiling
* Add runtime flag to enable profiling
* Add logic to run cpu profiling
* Update configuration parameter for profiling
* Add logic to enable memory profiling
* Update documentation (readme.md): see 'run' paragraph

* add more info in log (#713)

* fixed inconsistance block height on escrowed transaction (#717)

* Code Refactoring (#716)

* Use net/http/pprof package that serves the full profiling api via http
* Enable profiling api only when running app with -cpu-profile flag (see updated readme)
* Refactor the way monitoring server is instantiated: not using default http hanlder anymore
* Update readme.md

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#719)

* Refactor block query to select a range of blocsk instad of a single one for snapshots
* Refactor service logic to add more blocks (720) to snapshots

* Cleaning redundant function (#714)

* adding function for HD wallet testing

* cleaning some unused variables

* remove unused const and redundant function on generator

* Bug Fixing (#724)

* Fix pprof http server listening only on localhost

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#728)

* Optimize function GenerateTxRemoveNode

Co-authored-by: stefano galassi <[email protected]>

* 722 blockchain orchestrator (#726)

* add BlockchainOrchestrator to control the chains

* 709 Smithing All Blocksmith Skipped (#725)

* #709 add smithing pending blocker

* #709 remove smith time from blcoksmith model

* #709 update smithing process: recycle blocksmith when all blocksmith time are passed

* #709 separate smith time constants between spinechain and mainchain

* #709 inject spinechain blocksmith service

* #709 update metric smithtime to smith index

* #709 cover new smithing process unit tests

* #709 fix comment on blocksmith strategy & update fake smith function signature

* add index on block table and update sql connnection number (#729)

add index on block table and update sql connnection number (#729)

* scale down the common block jump (#731)

scale down the common block jump (#731)

* 721 - Remove Account Dataset Expiration (#732)

* accountDatasetsQuery should a plural name

* update affected services and more about account_dataset stuff

* fixing unit test of NewSnapshotFile integration

* escrow validation timeout limit exceeded

* Add node registry partial history to snapshots (#734)

* Code Refactoring

* Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process
* Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter
* Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve:
  * latest state from block 0 to 'fromHeight'
  * full history fromHeight to toHeight
  * merge and remove duplicate ids in first selection
* Remove unused constant LinkedReceiptBlocksLimit

Co-authored-by: stefano galassi <[email protected]>

* making the download process less restrictive to failures in catching  up with peers (#735)

* making the download process less restrictive to failures in catching up with peers

* fix the way of tracking errors

* fix the way of keeping track the error

* TestBlockIncompleteQueueService_AddTransaction (#720)

* TestBlockIncompleteQueueService_AddTransaction

* add succes case

* fix return type to use blocker (#742)

* fix return type to use blocker

* add more blocker type

* Update golangci to 1.24.0 (#743)

* update golangci to 1.24

* add reset db variable step

* fixed invalid space

* test circlleci building a branch

* fixes golangci warn and error

* disable ineffassign

* adjust the deployment step

* #740 index receipt fields for better query time (#744)

* #740 index receipt fields for faster query

* #740 verbose message on golangcilint run

* fix error casting (#747)

* fix error casting

* revert p2p error type

* fix blocker usage

* downgrade golangci-lint version (#748)

* Hotfix rows immediate release (#749)

* immediately release rows after converted to model

* add unique index to published_receipt.datum_hash

* fix: wrong column name

* temporary fix, omit the unique

* Unit Testing core service BlockPoolService (#739)

* NewBlockPoolService, GetBlocks, GetBlock, InsertBlock, ClearBlockPool

* add assertion

* gofmt -s -w

* Add optional Params on Sign function (#746)

* enhance sign function & cmd signature

* update readme

* add option params on GenerateSignedTxBytes

* update comment function SignEd25519

* add return err on parseBytesArgs func

* fix type & unused arg

* missing flag ed25519UseSlip10

* handle error from block push and validation (#751)

handle error from block push and validation (#751)

* fix block validation and smith time calculation (#752)

* setter and recipient must be the same person (#755)

* Fix: P2P - Can't Add UnresolvedPeers  (#757)

* add condition when there are still space in unresolved peers

* remove duplicate removal query (#758)

* Optimize database transaction (#756)

* use ExecuteSelectRow instead

* use QueryRow instead of Query, get single account balance

* use QueryRow instead if only want a single record

* nodeReg.ApplyConfirmed better to use QueryRow

* nodeRegUpdate.ApplyConfirmed better to use QueryRow

* removeNodeReg.ApplyConfirmed better to use QueryRow

* CMD fix and addition (#762)

* add blockheight logging at the end of applying a snapshot

* fix rollback cmd method to use the parameter passed

* add cmd methods for getting scrambledNodes and priorityPeers

* rename folder of scrambled nodes

* Use QueryRow instead of Query when getting single record (#764)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix query and the unit tests (#765)

fix query and the unit tests (#765)

* Confirm the state of download finish if the whole network stuck (#770)

* confirm the state of download finish if the whole network stuck

* reset the firstDownloadCounter when finding out they are not stuck

* reduce the validation limit constant

* Unit Test Transaction Core Service (#767)

* UndoApplyUnconfirmedTransaction

* ApplyConfirmedTransaction

* ApplyUnconfirmedTransaction

* ValidateTransaction

* remove underscores in Go names

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* GetBlockByHeight: Query row instead of Query (#768)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

* GetBlockByHeight better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix finish download (#771)

fix finish download (#771)

* reset db automatically and also snapshots stuff (#772)

* reset db automatically and also snapshots stuff

* *_kv would remove also

* snapshots suffix

* Fix manifest rollback (#773)

* add spineblock height field

* rename spine block manifest column and add indesx on block height column

* rename spine_block_manifest column field

* move spine_block_manifest to spinechain derived query category

* clean snapshot file on rollbacked

* add delete snapshot file function

* update tests

* remove ErrNoRows check as that error only occur on sql.Row not sql.Rows

* inject snapshot block service to spine block service

* snapshot manifest field rename

* fix new rollback query for manifest

* use parse instead of splitting chunks manually

* log the deletion error

* handle error

* update schema

* add extra comment for testing

* move file deletion to after db commit and as goroutine

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* #777 update validate block function signature : remove timestamp (#779)

* #777 update validate block function signature : remove timestamp

* move the validation of download finish if the block height is 0 (#781)

move the validation of download finish if the block height is 0 (#781)

* Bug limit receipt selection when select by root 774 (#776)

* versi 1

* small edit

* update mock ExecuteSelect

* make sure the query is similar as mock

* make it simpler

* merge schema

Co-authored-by: Nawi Kartini <[email protected]>

* 782 missing blocks fix (#783)

* #782 validate block in scanBlockPool thread

* #782 fix wrong function signature causing interface not implemented

* #782 move chain lock to beginning of ScanBlockPool function

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* Getting own public ip (#784)

* get own public ip

* allowing to get public ip via checkip.dyndns.org

* make sure the parsed ip is valid ip address

* IPUtil methods

* Fixing error nil pointer when accessing sql.Tx Executor (#786)

* add missing err checker on validate mempool

* fixing racing unlocking mutex & making sql.Tx nil

* Sort poppedBlocks by height ascending (#787)

* sort poppedBlocks by height desc

* fulfill test cases of PoppedoffBlocks

* sort ascending popped off blocks

* fulfill spine PopOffToBlock test cases

* #792 replace spendable with balance (#793)

* Genesis generate command: Code Refactoring and adding new feature to facilitate deployments with nomad/consul (#797)

* Code Refactoring and adding new feature

* Refactor command code to comply to new standards (see other commands)
* Add feature to generate consulKvInit.sh script from a template and a the newly generated genesis

* Bug fixing

* Limiter must be 2*MinRollbackHeight (#795)

* limiter must be 2*MinRollbackHeight

* 789 multisig pending transaction Apply-Flow (#798)

* #789 update pending_transaction query by hash to have multiple status filter

* #789 update flow of pending_transaction execution

* Tech: GetAccountBalances #785 (#788)

* first commit

* check num of addresses

* fix

* fix

* fix

* fix the unit testing

* use caseQuery instead of looping

* fix caseQuery.In

* fix unit testing

* move validation to handler

* fix unit test part 1

* auto update from proto

* accountBalance

* change to status.Error

* change to plural

* refine according PR comment

* fix

* fix unit test part 1

* remove commented line

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* fix the status for easier debug (#803)

fix the status for easier debug (#803)

* Bug: Limit Receipt Selection When Select By Root (#808)

* change 'where limit' on sql

* UnitTest-ApiService-GetBalances (#804)

* first commit (#807)

unit test on api handler `GetAccountBalance`

* #800 do mempool backup before executing derived query Rollback() method (#810)

* remove key from badger db after mempools backup restored (#811)

remove key from badger db after mempools backup restored (#811)

* add badger metrics to prometheus (#815)

* add badger metrics to prometheus

* explicitly use badger v2

* remove badgerdb v1

* revert MaxAPIRequestPerSecond to avoid nodes being overwhelmed by api calls

* enhance error logging for pushBlock and validateBlock process (#818)

* enhance error logging for pushBlock and validateBlock process

* exclude unimportant push and validate block error

* change block_height to reference_block_height (#820)

change block_height to reference_block_height (#820)

* Bug: Missing Rollback When BackupMempool Fail in PoppedOffBlock #802 (#816)

* first commit

* fix: Missing Rollback When BackupMempool Fail in PoppedOffBlock 

* change BackupMempols --> BackupMempools

* Unit test api handler account dataset (#821)

* GetAccountDatasets

* GetAccountDataset

* Expire musig pending transaction (#828)

* ignore .bak like genesis.go.bak :)

* expiring pending transactions by height and undo apply unconfirmed

* update readme about multis sig generator

* not allow filter by pending and expired

* finish query logic to, so now (block_height+min_rollback) = current_block_height

* defer func for rows close and matches the others about query logic

* add log instance into transaction core service, ignore log level

Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* Unit test api block handler (#830)

* TestBlockHandler_GetBlock

* TestBlockHandler_GetBlocks

* fix captLocal

* fix gocritic

* Unit test api mempool handler (#831)

* TestMempoolTransactionHandler_GetMempoolTransaction

* TestMempoolTransactionHandler_GetMempoolTransactions

* 829 download escrow transaction (#834)

* #832 add one more persistance time cycle to check to avoid node stuck in small network

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #833 add copy of block to broadcast before deleting the transactions

* #829 fix test

* #829 add test for escrow transaction query

* #833 broadcasted wrong variable

* #829 refactor escrow transaction query to be more generic

* Hotfix deepcopy broadcast block (#837)

* add mohae/deepcopy dependency

* enforce deep copy for broadcast block

* use map to reduce loop complexity (#838)

* get the latest account balances (#839)

Co-authored-by: iltoga <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Kevin Hadi <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
iltoga added a commit that referenced this pull request Jun 9, 2020
* Code Refactoring (#587)

* Add spine blockquery to blocksmithstrategyspine to be able to get the previous spine block when selecting/sorting blocksmiths

* Get escrow transactions with filter fields and pagination (#590)

* get escrow transactions with filter fields and pagination

* get escrow transactions functional test

* #588 decouple published receipt and participation score from block main service (#591)

* 592 fee model (#595)

* #592 add fee model to action type

* #592 add fee model to action type

* #592 ignore package-interface naming lint rule

* #592 update pb file of transaction : add model + service for getTxMinimumFee api

* #592 add handler for calculate minimum fee api

* #592 fix GetTransactionMinimumFee request

* #592 fix wrong type

* #592 add api client for testing get minimum fee

* 578 snapshot file creation (#593)

* Logic for snapshot file creation

* Add new interface and service to abstract snapshot file creation for different block types (for now only mainchain)
* Add ugorji/go/codec package to add CBOR serialization support

* Added Logic for snapshot file processing (create/download)

* Add new interface and service to abstract snapshot file download
* Implemented some of the logic in snapshotMainBlockService for downloading and creating new snapshot files/chunks

* Added business logic for snapshots

* Add new service FileService to abstract (and make testable) methods relative to file manipulation

* Better separation of logic between shapshotservice (generic methods) and snapshotBlockService (blocktype-specific methods)
* Move GetFileNameFromHash and GetHashFromFileName into fileservice
* Add query and service handler to retrieve account balances for snapshots
* Add minimal process management for snapshot files: if a snapshot of the same block type is still running and a new one is started, the new one will stop the old one (this logic must be discussed and is a WIP)
* Add initial logic to create a snapshot file containing all (only last state) account balances
* TODO: increase code coverage (few methods are tested so far)

* Bug fix

* Add EscrowTransactionQuery to GetDerivedQueries

* Added logic for generating snapshot files

* Add All relevant queries and wrappers (to ease testing) to snapshotMainBlockService
* Add test case for NewSnapshotFile and new queries

* Finalised logic for generating snapshot files (first iteration only has single file per snapshot)

* Add all relevant queries to get snapshot's payload
* Add method to parse and import snapshot into db
* Add config parameter for snapshots folder
* Add unit and integration tests

* Added logic for download snapshots

* Add method to parse the spineBlockManifest and download file chunk hashes
* TODO: download file is still mocked

* Uncomment logic previously commented for testing purposes
* Increase MainchainSnapshotInterval value to 1440 to avoid having too many snapshots in dev and production environments

* Add query interface for snapshot-specific queries (SnapshotQuery)
* Add SelectDataForSnapshot interface signature to new interface
* Add GetSnapshotQuery method to get all snapshot queries implementations (in a map)
* Refactor relative query methods in every relevant repository
* Refactor some constant to use time.Duration instead of int type
* Fix unit tests

* Add DESC to order by clauses in snapshot queries
* Refactor GetChainTypeCount to GetChainTypes to make chain types countable
* Remove temporary test function


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)


* Use SnapshotQuery interface to loop through all snapshot query implementations instead of adding them one by one (less code duplication)

* 596 multisig skeleton (#597)

* #596 add multisignature transaction body

* #596 add constants for MultiSignature body fields

* #596 add multisignature tx type skeleton

* #596 add multisig tx type enum

* #596 add command tools for generate multisignature tx and parsing any tx

* #596 basic multisignature parsing functionality

* #596 update schema commit

* #596 add signature info

* #596 update signature structure to use map

* #596 adjust cmd tools to new multisig byte structure

* #596 update multisig transaction bytes

* #596 update schema

* 599 snapshot chunk strategy (#605)

* Added proto message for snapshots

* Add SnapshotPayload message representing internal snapshot data structure

* Added logic for snapshot chunk 'basic' strategy

* Add Strategy (interface and implementation) for 'basic' snapshot chunks: chunks are generated by splitting a full encoded snapshot
* Add SnapshotPayload model to schema, since we are using it in two different packages
* Refactor snapshotmainblockservice to use new snapshot chunk strategy
* Refactor unit and integration tests

* Completing logic for snapshot basic chunk strategy

* Finalize BuildSnapshotFromChunks and its test
* Add ReadFileByHash to FileService to be able to directly read a file from its hash-name and mock ioutil.ReadFile in services that depend on this service

* Remove unused parameter 'chunkSizeBytes' from SnapshotBlockServiceInterface.NewSnapshotFile

* 600 multisig queries (#607)

* #600 add new managed table for multisig

* #600 update multisig managed table model

* #600 add managed table query for multisig

Co-authored-by: Ariasa Ketut <[email protected]>

* 602 generate multisig address (#609)

* method multisig account address assembler

* command line tool multisig generator command

* set 1.20.0 of golangci-lint version to fixed circleci fail

* no need to validate current block height (#610)

* inhibit spine blocks smithing process during main blocks first download (#608)

* Refactor spineBlockDownloadService to blockStatusService, to be able to set/get all chaintype blockchain statuses (for now only if a node has finished downloading the block first time after has started)
* Refactor downloadBlockchain to use blockStatusService as dependency and set blockchain download status referenced by blockStatusService
* Simplify main.go by moving logic to set first blockchain download into downloadBlockchain service
* Add new blocker type P2PNetworkConnectionErr
* Increase timers for checking block download process
* Refactor blockStatusService and implementations to blockTypeStatusService
* Refactor relative code in other services
* Remove cast to int snapshotPayloadHeight in snapshotMainBlockService
* Fix bug in snapshotPayloadHeight computation
* Refactor constructor method for BlockchainSyncService

* Fixed transaction generator affected by fee (#611)

* no need to validate current block height

* fixed transcation generator afftected by fee

* Code Refactoring and Bug Fixing (#613)

* Fix logic for unlocking smithing process based on network status and blockchain download status (all main blocks downloaded)
* Add new service variable and relative getter/setter to blockTypeStatusService 'isSmithingLocked' to control whether smithing is locked or not, regardless of the blockchain download status
* Remove unused getter from chainType
* Add new constants for better code readability
* Fix tests

* 601 multisig validation (#615)

* #601 add --post and --post-host to enable post transaction bytes on generated

* #601 add multisig validation

* #601 fix lint

* #601 wrong description in cmd

* Optimize GRPC duration (#616)

* no need to validate current block height

* set index into some tables for speed up latency get a lot of records

* remove unused a line of codes

* GRPC endpoint get single escrow transaction by ID (#620)

* no need to validate current block height

* functionality test get escrow by id

* Download snapshot chunks (#614)

* Change fatalf to errorf/return in goroutine in snapshotService.go
* Add missing return statement for error in snapshotService.go

* Added logic for downloading files from a random peer and Code Refactoring

* Refactor filedownloader: moved to p2p package because being in service generates a cycle dependecy
* Add rpc call (client request and server response) to download a file from a random peer
* Add logic to download/parse/verify a file and a snapshot

* Added logic and file download metrics

* Add logic to download a file, server side
* Add metrics to count succeeded/failed snapshot downloads

* Update schema: add Failed files for file download
* Add metric IncrementSnapshotDownloadCounter to count number of downloaded files from every node
* Refactor InsertSpineBlockManifest to not fail if a new record with same id is inserted, to allow downloaded spine block manifests to override locally (probably wrong) generated ones
* Rename megablockQuery to spineBlockManifestQuery
* Refactor blockchainStatusService to use safer (for r/w race conditions) and faster sync.Map instead of regular map + sync.mutex
* Fix bug in spineBlockService.push: insert spineBlockManifest records if they are present as part of block data
* Fix smith process lock when downloading spine and main blocks
* Fix download blockchain process: spine blocks, snapshot, rest of main blocks. then start smithing


* Add retry download failed file chunks from different peers
* WIP: commented code to make it easier for others to run test nodes locally and double check the download process works as expected. All commented code is prepend with "// STEF" string

* Simplified fileService GetFileNameFromHash and GetFileNameFromBytes so that they don't have to return error and refactored relative code that uses them
* Add basic unit test for p2p.DownloadFilesFromPeer (still needs more test cases)
* Optimize p2p.DownloadFilesFromPeer while writing the test
* Fix code after code review

* fixed bug in caclulating snapshot's height

* Add two struct and methods to mimic sync.Map behaviour, but with regular map + sync.RWMutex (more efficient on machines with low cpu core count)
* Refactor code that used sync.Map in favor of new map struct
* Add util function to calculate a quick (non secure) random number
* Fix uncaught nil in p2pServerHandler

* Add NewSpineBlockManifestQuery to derivedqueries to make it rollback safe (it must be rolledback together with mainchain blocks)
* Add a couple of TODOs for spine block manifest validation

* Fix bug in unitialized model.MapIntBool struct
* Add missing test for GenerateSnapshot method

* Uncommented code for testing

* Uncomment code
* Fix one unit test after fixing the snapshot height (previous commit)

Co-authored-by: stefano galassi <[email protected]>

* 594 add bitcoin signature (#618)

* add bitcoin sig, btcd &  btcutil package

* restructure Ed25519 signature type function

* add cmd to generate bitcoin account

* add signature-type flag in cmd tx generator

* cmd enhace switch case generate account

* return error when sign and verify payload

* cmd generate tx with sender signature type

* adjust general parse transaction bytes

* adjust unit test

* udate cmd readme

* cmd, panic when invalid signature type

* remove unused function

* add generate account in signature

* add signature unit test

* add misisng import package

* update schema

* change parameter type of GenerateAccountFromSeed

Co-authored-by: Ariasa Ketut <[email protected]>

* 629 - Use gomod consistently (#630)

* no need to validate current block height

* update circleci config use go modules instead of dep and update go version to 1.14

* go mod and more cleaning shadow declaration variable

* Revert "GRPC endpoint get single escrow transaction by ID (#620)"

This reverts commit 7824924.

* Revert "Revert "GRPC endpoint get single escrow transaction by ID (#620)""

This reverts commit bedbf87.

* revert unused commit

* remove Gopkg.* files

* remove no needed comment

* make sure go modules cache correctly

* change circleci working directory, update readme

* update path of cache

* fixed go modules caches

* 603 multisig apply (#623)

* #603 initialize multisig service layer

* #603 implement apply confirmed

* #603 add sender address in pending transaction

* #603 update multisignature.PendingTransaction schema

* #603 delete multisig tx util

* #603 add sign command to support multisig development

* #603 add --sender-address flag for generating unsigned bytes

* #604 add sender address to unsigned tx

* #603 return raw error from execute transaction to enable sqlite error casting

* #603 complete multisig implementation

* #603 fix tests

* #603 add latest flag to pending transaction

* #603 update pending_transaction status on executed

* #603 add latest flag to pending transaction

* #603 add filter to pending transaction query

* #603 add latest flag to multisig-info and signatureInfo

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx table

* #603 add multisig-child flag to tx model

* #603 add latest field to all pending tables

* #603 adjust query to handle latest flag

* #603 insert multisig-child transaction to transaction table

* #603 dfix test

* #603 adjust cmd tools and crypto package to validate multisig signature (hierarchical multisig)

* #603 fix example cmd

* #603 update schema

* #603 remove binary

* #603 convert address-Signature to format stringTostring

* #603 fix query inconsistency

* #603 handle unsigned transaction execution status

* #603 revert unnecessary change

* #603 handle submit multisig info execute multiple tx

* #603 fix query inconsistency

* #603 add multisig signature type

* #603 add constant for multisig signature validation type;

* #603 remove debug log

* CI delivery image version (#632)

* fixed invalid config

* change image version to stretch

* remove configuration for test purposes

* Ci delivery image version (#633)

* fixed typo

* 604 api rate limit (#625)

* add rate limiting feature for API

* fix unclosed rows

* change MaxAPIRequestPerSecond from int32 to uint32

* add download process metrics tracker

* add go sum

* 621 - Respecting  escrow expiration on Push Block (#626)

* no need to validate current block height

* fixing bug, respecting expiring escrow process on PushBlock

* fixed escrow rollback query and respecting escrow stage before push block process

* fixed escrow unit test result

* has been fixed forgot to close rows

* allow expiring escrow method to control db transaction

* Spine block manifest and spine public keys validation when downloading blocks from peers (#631)

* Bug fixing

* Add (dummy) ValidatePayloadHash to be used to validate payload data when downloading a block + attached data from other peers
* Add GetPayloadHash and separated logic to compute block's payload hash from GenerateBlock to an individual method to make it reusable (eg. in GenerateBlock and ValidatePayloadHash)
* Fix tests

* Code Refactoring

* Refactored method's name from GetPayloadHash to GetPayloadHashAndLength

* Added logic to validate block's payload on block push

* Add ValidatePayloadHash logic to blockMainService and blockSpineService
* Fix tests

* Add logic to validate spine block manifest against its relative spine block of reference

* Add GetBlockFromTimestamp to blockQuery to be able to get the spine block of reference for a spine block manifest (which is the first block with timestamp >= spineBlockManifest.ExpirationTimestamp)
* Add blockSpineService-specific method ValidateSpineBlockManifest to validate a spine block manifest after is downloaded from the p2p network
* Add unit tests for new methods

* Merge remote-tracking branch 'origin/develop' into 627-spine-block-manifest-publickeys-validation

# Conflicts:
#	core/service/blockMainService.go

* Code Refactoring

* Update genesis block ID after changes to Generate block (added published receipts to payload length)
* Add unit test for validateblockhash

* Code Refactoring

* Add 'dummy' interface BlockServiceSpineInterface to collect all signatures of spine block-specific methods

* Code Refactoring

* Removed unnecessary comment

* Bug fixing

* Remove fmt.println probably left from some internal test

* Bug fixing

* Update comment in query

* Code Optimization

* Optimize if statement by reversing conditions (courtesy of @ali)

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#636)

* Fixed bug in download snapshot process causing segfault

Co-authored-by: stefano galassi <[email protected]>

* Bug Fixing (#639)

* Fix bug in pendingSignatureQuery rollback function
* Remove unused function from transactionQuery

Co-authored-by: stefano galassi <[email protected]>

* Escrow query test fail fixed (#640)

* no need to validate current block height

* escrow query test fail fixed

* Bug Fixing (#642)

* Fix bug in unchecked map key leading to panic in block smith process

* Escrow query test fail fixed (#647)

* no need to validate current block height

* escrow query test fail fixed

* fixed escrow map params need to sorting before compare

* 634 multisig api (#645)

* #634 init multisig api

* #634 update multisig request model

* #634 update get pending tx detail response

* #634 update get pending tx detail response

* #634 initiate multisig client API

* #634 fix bug: wrong empty signature checks

* #634 remove built binary from repository

* #634 add multisig api tests

* #634 revert unnecessary changes

* #634 update go mod

* #634 initiate health check service and model

* #646 add health check endpoint

* #634 update schema

* #634 use getter instead of accessing field directly

* #634 add stricter param check

* #634 fix pagination

* #634 remove duplicate code

Co-authored-by: Ariasa Ketut <[email protected]>

* 634 multisig api (#653)



* #634 add default field to order field param

Co-authored-by: Ariasa Ketut <[email protected]>

* #651 fix multisig_address signature validation (#652)

* #651 fix multisig_address signature validation

* #651 fix test

Co-authored-by: Ariasa Ketut <[email protected]>

* 656 incorrect account ledger balance changes (#657)

* no need to validate current block height

* account ledger incorrect balance change information fixed

* 654 account ledger multisig (#658)

* #654 add multisig transaction event to account ledger

* #654 add ledger record in multisig transaction

* #654 update schema commit

Co-authored-by: Gede Sukra Widhyawan <[email protected]>

* 660 api get multisig info (#661)

* #660 add new api endpoint proto for getting multisignature info

* #660 implement get multisig info api

* #660 rename getPendingTransactions request

* #660 rename getPendingTransactions request

* #660 update schema commit

* 650 - CMD HD wallet generate account (#668)

* no need to validate current block height

* generate account slip10

* allow fetch from private repo

* fixed invalid format of ci

* fixed set env of machine

* add additional ssh key

* fixed bash condition when check gomod exists

* allowed github connection via token instead of ssh

* remove unused code

* more describe generate account command

* fix uninitialized metrics (#662)

fix uninitialized metrics (#662)

* bitcoin sig: add public key into signature (#670)

* bitcoin sig: add public key into signature

* handle multiple rows in a function (#675)

handle multiple rows in a function (#675)

* SelectReceipt Limit (#676)

SelectReceipt Limit (#676)

* Circleci GitHub token (#677)


* token as env variable

* 628 apply snapshot (#667)

* added alpha nodes genesis.go

* Rollback mempool - Restoring mempool transactions backup (#419)

* initiate restoreMempoolsBackup func

* finished mempool backup process and add loggger into processfork struct

* add logger into blockPopper stuff

* 420 get scramble nodes enhance (#423)

* #420 rename get scrambled nodes function to get last scrambled node, since we will have multiple version now

* #420 move some scrambling function to util to reduce fetching function call frequency

* #420 add block_height to merkle tree table

* #420 fix query to fetch receipt, it had wrong block height scope on the query

* #420 handle error

* #420 add receipt validation

* #420 build scrambled node at height x

* #420 adjust constructor update

* #420 move receipt service instantiation to after node registration service

* #420 move scramble node model outside, it is used by other service as well : receipt service

* #420 validate receipt before storing to database

* #420 use bytes.equal for better comparison performance

* #420 rename memoized scramble nodes

* #420 simplify loop

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted broken query and kept changes from newer pr

* 432 revert participation score query (#437)

* 424 fix add participation score query (#425)

* fixed addParticipationScore query
refactored unit test
refactored nodeRegistration applyConfirmed to use addParticipationScore instead of insertParticipationScore for gnenesis block (so we can remove that unsafe query :) )

* removed unused InsertParticipationScore query

* refactored addparticipationscore query to use a single upsert statement

* Remove some process (#429)

* disable some process to adjust with alpha
* update unit test

* reverted query

* reverted broken participation score query

* change a few configurations special for alpha (#434)

* change a few configurations special for alpha

* updated config.toml

* added (peer) port to genesis nodes addresses

* fixed test

* updated production values for some constants

* updated log levels for testing alpha nodes

* reset default log levels

* increased PriorityStrategyBuildScrambleNodesGap to 40

* fixed unwanted diffrences between develop and alpha branches

* fixed unit tests for alpha

* fixed unit test for mempoolCoreService

* Fixed unit test

* Commented out startSpinechain, to inhibit spine blocks generation: at this stage it won't make any difference, since we need snapshots to make a use of spine blocks

* switch log level for debugging purpose

* debug: process fork warn to errorf log

* revert: revert log levels

* keep track of p2p go routine metrics (#569)

* fix shadowed error message

* Resolved merge conflicts

* Code refactoring

* Updated mainchain genesis id

* Code Refactoring

* Update main block genesis ID

* Updated constant

* Update constant DownloadSnapshotNumberOfRetries to be = MaxResolvedPeers so that if a file fails to download the node will cycle trough all resolved peers until it is downloaded or it fails

* Update golang.org/x/crypto/ed25519 to crypto/ed25519

* Reverted crypto/ed25519 to golang.org/x/crypto/ed25519

* Fixed bug in download snapshot process causing segfault

* Adding business logic

* Add SnapshotQueries for pendingsignature, pendingtransaction, multisiginfo
* Add logic to snapshot service to include multisig data in snapshots


* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries

* Added business logic

* Add TrimDataBeforeSnapshot to all SnapshotQueries
* Refactor snapshot file downloader to return snapshotFileInfo (ready to be read/imported)
* Add logic to main.go to import snapshot after downloading it

* Refactor IsSnapshotHeight with simplified logic

* Refactor code to be aligned with develop branch instead of alpha

* Refactor SelectDataForSnapshot queries to use rollback-like logic

* Refactor SelectDataForSnapshot queries to use rollback-like logic
* Write unit tests for new queries

* Fix wrong table nale in new block query

* Fix wrong table nale in new block query

* Fix do not allow to delete genesis block when trimming block table

* Revert configX.toml log level to "fatal", "error", "panic"

* #628 add checks if all blocksmith time has expired

* #628 move push block to anonymous function

* Add rollback to snapshot height to set latest = 1 to all latest entitites

* Fixed SelectDataForSnapshot query in accountDataset repo

* Fix missing for loop break conditions in ScanBlockPool
* Add function to log running binary file checksum at node startup

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: stefano galassi <[email protected]>

* 672 makefile binary built (#678)

* #672 ignore release folder

* #672 initialize makefile

* #672 add github token example file

* #672 move xgo to zoobc repository

* #672 add makefile usage

* #672 add cc for cmd

* #672 add go-fmt and separate golangcilint version

* #672 move test to makefile command

* #672 fix gopath

* #672 update readme for github.token usage

* #672 print information for step ran

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* 665-Enhance CMD Database Init (#680)

* cmd: put init db only where is needed

* cmd: add commad to generate proof of ownership

* cmd: add readme rollback

* cmd: fixing remove node reference func

* handle nil value (#682)

* throw error on over limit get blocks (#684)

* throw error on over limit get blocks

* move limit validation to handler layer

* remove pending tx status change, it has been updated in apply confirmed (#685)

* 686: Cmd, enhance generate account (#687)

* cmd: enhance generate account

* add get public key using slip10

* 655 escrow receipt approval (#688)

* no need to validate current block height

* account dataset escrow approval and integration

* fullfill send money validation test cases

* to be gentle no need to check the account dataset property

* create specific method of query for account dataset

* getHostInfo unit testing (#689)

* getHostInfo unit testing
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* adding database status metrics (#693)

adding database status metrics (#693)

* Bug Fixing snapshot selection queries and enhancing rollback queries (#696)

* Bug Fixing

* Fix snapshot selection queries
* Enhance rollback queries
* Enhance db tables by adding composite primary keys where possible
* Fix bug in rollback and snapshot selection queries

Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 681 disable smithing non registered node (#694)

* #681 add zero participation score blocker

* #681 inject blockchainStatusService to noderegistration service

* #681 add flag to say if a node is blocksmith or not

* #681 update main.go with new struct constructor

* #681 only update the blocksmith status when detect zero participation score

* #681 compare with bytes.equal()

* unit testing for GetBlockQueue (#698)

unit testing for GetBlockQueue (#698)

* Bug Fixing (#706)

* Fix breaking bug in rollback query
* Refactor spinePublicKey rollback query to comply to new code logic

* 691 grpc account dataset (#699)

* no need to validate current block height

* rename accountDataset to plural name

* account dataset grpc service list and single record

* grpc client example

* move params checkup to handler package

* remove unnecessary test cases

* technical error message passed

* 690 empty block smith (#703)

* #690 add empty flag to enable empty block generate

* #690 add limit for skipped block smith

* #690 set the skipped limit back to 15

* #690 remove unimplemented test

* #690 fix return arguments

* #690 remove unnecessary variable initialization

* #690 make use of default int64 value

* #690 smaller skipped blocksmith constant so we can see the effect on dev net

* adding function for HD wallet testing (#683)

* adding function for HD wallet testing

* P2P: Add unit test on server service (#707)

* p2p: add unit test p2p server service

* enhance Err format

* change GetPeerInfoResponse field name

* p2p: update unit test

* update schema

Co-authored-by: Andy Shi <[email protected]>

* added skippedblocksmith to snapshot (generate/apply) (#710)



* Add SkippedBlocksmith to snapshot proto model

* Added Logic for snapshots

* Add selectDataForSnapshot and TrimDataBeforeSnapshot queries to skippedBlockSmithQuery repository

* Added Logic for snapshots

* Add logic to include skippedblocksmith records to snapshots
* Add logic to import skippedblocksmith records from snapshots into db (apply snapshot)
* Fix unit tests

* Use getter method instead of direct variable in skippedBlocksmithQuery TrimDataBeforeSnapshot

Co-authored-by: stefano galassi <[email protected]>

* Adding runtime cpu profiling for core node (#712)

* Adding Application Logic
* Add configuration parameter for profiling
* Add runtime flag to enable profiling
* Add logic to run cpu profiling
* Update configuration parameter for profiling
* Add logic to enable memory profiling
* Update documentation (readme.md): see 'run' paragraph

* add more info in log (#713)

* fixed inconsistance block height on escrowed transaction (#717)

* Code Refactoring (#716)

* Use net/http/pprof package that serves the full profiling api via http
* Enable profiling api only when running app with -cpu-profile flag (see updated readme)
* Refactor the way monitoring server is instantiated: not using default http hanlder anymore
* Update readme.md

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#719)

* Refactor block query to select a range of blocsk instad of a single one for snapshots
* Refactor service logic to add more blocks (720) to snapshots

* Cleaning redundant function (#714)

* adding function for HD wallet testing

* cleaning some unused variables

* remove unused const and redundant function on generator

* Bug Fixing (#724)

* Fix pprof http server listening only on localhost

Co-authored-by: stefano galassi <[email protected]>

* Code Refactoring (#728)

* Optimize function GenerateTxRemoveNode

Co-authored-by: stefano galassi <[email protected]>

* 722 blockchain orchestrator (#726)

* add BlockchainOrchestrator to control the chains

* 709 Smithing All Blocksmith Skipped (#725)

* #709 add smithing pending blocker

* #709 remove smith time from blcoksmith model

* #709 update smithing process: recycle blocksmith when all blocksmith time are passed

* #709 separate smith time constants between spinechain and mainchain

* #709 inject spinechain blocksmith service

* #709 update metric smithtime to smith index

* #709 cover new smithing process unit tests

* #709 fix comment on blocksmith strategy & update fake smith function signature

* add index on block table and update sql connnection number (#729)

add index on block table and update sql connnection number (#729)

* scale down the common block jump (#731)

scale down the common block jump (#731)

* 721 - Remove Account Dataset Expiration (#732)

* accountDatasetsQuery should a plural name

* update affected services and more about account_dataset stuff

* fixing unit test of NewSnapshotFile integration

* escrow validation timeout limit exceeded

* Add node registry partial history to snapshots (#734)

* Code Refactoring

* Add GetBlocksmithSafeQuery to query.go to retrieve a map with all queries that must save partial history to snapshots to not break blocksmith process
* Refactor NewSnapshotFile (snapshot creation function) to use GetBlocksmithSafeQuery and get correct snapshot 'fromHeight' parameter
* Refactor nodeRegistrationQuery 'SelectDataForSnapshot' to retrieve:
  * latest state from block 0 to 'fromHeight'
  * full history fromHeight to toHeight
  * merge and remove duplicate ids in first selection
* Remove unused constant LinkedReceiptBlocksLimit

Co-authored-by: stefano galassi <[email protected]>

* making the download process less restrictive to failures in catching  up with peers (#735)

* making the download process less restrictive to failures in catching up with peers

* fix the way of tracking errors

* fix the way of keeping track the error

* TestBlockIncompleteQueueService_AddTransaction (#720)

* TestBlockIncompleteQueueService_AddTransaction

* add succes case

* fix return type to use blocker (#742)

* fix return type to use blocker

* add more blocker type

* Update golangci to 1.24.0 (#743)

* update golangci to 1.24

* add reset db variable step

* fixed invalid space

* test circlleci building a branch

* fixes golangci warn and error

* disable ineffassign

* adjust the deployment step

* #740 index receipt fields for better query time (#744)

* #740 index receipt fields for faster query

* #740 verbose message on golangcilint run

* fix error casting (#747)

* fix error casting

* revert p2p error type

* fix blocker usage

* downgrade golangci-lint version (#748)

* Hotfix rows immediate release (#749)

* immediately release rows after converted to model

* add unique index to published_receipt.datum_hash

* fix: wrong column name

* temporary fix, omit the unique

* Unit Testing core service BlockPoolService (#739)

* NewBlockPoolService, GetBlocks, GetBlock, InsertBlock, ClearBlockPool

* add assertion

* gofmt -s -w

* Add optional Params on Sign function (#746)

* enhance sign function & cmd signature

* update readme

* add option params on GenerateSignedTxBytes

* update comment function SignEd25519

* add return err on parseBytesArgs func

* fix type & unused arg

* missing flag ed25519UseSlip10

* handle error from block push and validation (#751)

handle error from block push and validation (#751)

* fix block validation and smith time calculation (#752)

* setter and recipient must be the same person (#755)

* Fix: P2P - Can't Add UnresolvedPeers  (#757)

* add condition when there are still space in unresolved peers

* remove duplicate removal query (#758)

* Optimize database transaction (#756)

* use ExecuteSelectRow instead

* use QueryRow instead of Query, get single account balance

* use QueryRow instead if only want a single record

* nodeReg.ApplyConfirmed better to use QueryRow

* nodeRegUpdate.ApplyConfirmed better to use QueryRow

* removeNodeReg.ApplyConfirmed better to use QueryRow

* CMD fix and addition (#762)

* add blockheight logging at the end of applying a snapshot

* fix rollback cmd method to use the parameter passed

* add cmd methods for getting scrambledNodes and priorityPeers

* rename folder of scrambled nodes

* Use QueryRow instead of Query when getting single record (#764)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix query and the unit tests (#765)

fix query and the unit tests (#765)

* Confirm the state of download finish if the whole network stuck (#770)

* confirm the state of download finish if the whole network stuck

* reset the firstDownloadCounter when finding out they are not stuck

* reduce the validation limit constant

* Unit Test Transaction Core Service (#767)

* UndoApplyUnconfirmedTransaction

* ApplyConfirmedTransaction

* ApplyUnconfirmedTransaction

* ValidateTransaction

* remove underscores in Go names

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* GetBlockByHeight: Query row instead of Query (#768)

* removeNodeReg.Validate better to use QueryRow

* nodeRegistrationUpdate.ApplyUnconfirmed better to use QueryRow

* GetBlockByHeight better to use QueryRow

Co-authored-by: Andy Shi <[email protected]>

* fix finish download (#771)

fix finish download (#771)

* reset db automatically and also snapshots stuff (#772)

* reset db automatically and also snapshots stuff

* *_kv would remove also

* snapshots suffix

* Fix manifest rollback (#773)

* add spineblock height field

* rename spine block manifest column and add indesx on block height column

* rename spine_block_manifest column field

* move spine_block_manifest to spinechain derived query category

* clean snapshot file on rollbacked

* add delete snapshot file function

* update tests

* remove ErrNoRows check as that error only occur on sql.Row not sql.Rows

* inject snapshot block service to spine block service

* snapshot manifest field rename

* fix new rollback query for manifest

* use parse instead of splitting chunks manually

* log the deletion error

* handle error

* update schema

* add extra comment for testing

* move file deletion to after db commit and as goroutine

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* #777 update validate block function signature : remove timestamp (#779)

* #777 update validate block function signature : remove timestamp

* move the validation of download finish if the block height is 0 (#781)

move the validation of download finish if the block height is 0 (#781)

* Bug limit receipt selection when select by root 774 (#776)

* versi 1

* small edit

* update mock ExecuteSelect

* make sure the query is similar as mock

* make it simpler

* merge schema

Co-authored-by: Nawi Kartini <[email protected]>

* 782 missing blocks fix (#783)

* #782 validate block in scanBlockPool thread

* #782 fix wrong function signature causing interface not implemented

* #782 move chain lock to beginning of ScanBlockPool function

Co-authored-by: Ahmad Ali Abdilah <[email protected]>

* Getting own public ip (#784)

* get own public ip

* allowing to get public ip via checkip.dyndns.org

* make sure the parsed ip is valid ip address

* IPUtil methods

* Fixing error nil pointer when accessing sql.Tx Executor (#786)

* add missing err checker on validate mempool

* fixing racing unlocking mutex & making sql.Tx nil

* Sort poppedBlocks by height ascending (#787)

* sort poppedBlocks by height desc

* fulfill test cases of PoppedoffBlocks

* sort ascending popped off blocks

* fulfill spine PopOffToBlock test cases

* #792 replace spendable with balance (#793)

* Genesis generate command: Code Refactoring and adding new feature to facilitate deployments with nomad/consul (#797)

* Code Refactoring and adding new feature

* Refactor command code to comply to new standards (see other commands)
* Add feature to generate consulKvInit.sh script from a template and a the newly generated genesis

* Bug fixing

* Limiter must be 2*MinRollbackHeight (#795)

* limiter must be 2*MinRollbackHeight

* 789 multisig pending transaction Apply-Flow (#798)

* #789 update pending_transaction query by hash to have multiple status filter

* #789 update flow of pending_transaction execution

* Tech: GetAccountBalances #785 (#788)

* first commit

* check num of addresses

* fix

* fix

* fix

* fix the unit testing

* use caseQuery instead of looping

* fix caseQuery.In

* fix unit testing

* move validation to handler

* fix unit test part 1

* auto update from proto

* accountBalance

* change to status.Error

* change to plural

* refine according PR comment

* fix

* fix unit test part 1

* remove commented line

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* fix the status for easier debug (#803)

fix the status for easier debug (#803)

* Bug: Limit Receipt Selection When Select By Root (#808)

* change 'where limit' on sql

* UnitTest-ApiService-GetBalances (#804)

* first commit (#807)

unit test on api handler `GetAccountBalance`

* #800 do mempool backup before executing derived query Rollback() method (#810)

* remove key from badger db after mempools backup restored (#811)

remove key from badger db after mempools backup restored (#811)

* add badger metrics to prometheus (#815)

* add badger metrics to prometheus

* explicitly use badger v2

* remove badgerdb v1

* revert MaxAPIRequestPerSecond to avoid nodes being overwhelmed by api calls

* enhance error logging for pushBlock and validateBlock process (#818)

* enhance error logging for pushBlock and validateBlock process

* exclude unimportant push and validate block error

* change block_height to reference_block_height (#820)

change block_height to reference_block_height (#820)

* Bug: Missing Rollback When BackupMempool Fail in PoppedOffBlock #802 (#816)

* first commit

* fix: Missing Rollback When BackupMempool Fail in PoppedOffBlock 

* change BackupMempols --> BackupMempools

* Unit test api handler account dataset (#821)

* GetAccountDatasets

* GetAccountDataset

* Expire musig pending transaction (#828)

* ignore .bak like genesis.go.bak :)

* expiring pending transactions by height and undo apply unconfirmed

* update readme about multis sig generator

* not allow filter by pending and expired

* finish query logic to, so now (block_height+min_rollback) = current_block_height

* defer func for rows close and matches the others about query logic

* add log instance into transaction core service, ignore log level

Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* Unit test api block handler (#830)

* TestBlockHandler_GetBlock

* TestBlockHandler_GetBlocks

* fix captLocal

* fix gocritic

* Unit test api mempool handler (#831)

* TestMempoolTransactionHandler_GetMempoolTransaction

* TestMempoolTransactionHandler_GetMempoolTransactions

* 829 download escrow transaction (#834)

* #832 add one more persistance time cycle to check to avoid node stuck in small network

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #829 include escrow detail of transaction in GetTransactionByBlockID

* #833 add copy of block to broadcast before deleting the transactions

* #829 fix test

* #829 add test for escrow transaction query

* #833 broadcasted wrong variable

* #829 refactor escrow transaction query to be more generic

* Hotfix deepcopy broadcast block (#837)

* add mohae/deepcopy dependency

* enforce deep copy for broadcast block

* use map to reduce loop complexity (#838)

* get the latest account balances (#839)

* Unit test api node admin handler (#840)

* TestNodeAdminHandler_GetProofOfOwnership

* TestNodeAdminHandler_GenerateNodeKey

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* 846 escrow inconsistent (#847)

* #846 add query to fetch expired escrow transaction at given height

* #846 save escrow_transaction.timeout as it is (submitted by client / node)

* #846 modify expiring escrow transaction function to use the new query

Co-authored-by: Ariasa Ketut <[email protected]>

* RequestBlockTransactions must include escrow part (#848)

* now requesting block transactions will include escrows

* inconsistence indexing of map test cases fixes

* might be only pending ecrow tx included

* Bug: Error RESOURCE_EXHAUSTED GetTransactions (#849)

* Pagination can't be empty

* set default pagination

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* Unit test api multisig handler (#841)

* TestMultisigHandler_GetPendingTransactions

* TestMultisigHandler_GetPendingTransactionDetailByTransactionHash

* TestMultisigHandler_GetMultisignatureInfo

* fix error ci/circleci

* fix error ci/circleci

* remove unuse print

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>
Co-authored-by: Andy Shi <[email protected]>

* Unit Test Api TransactionHandler (#850)

* first commit

* fix misspelling and too long line of codes

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* first commit (#851)

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* Re-ApplyUnconfirmed pending transactions (#853)

* re apply unconfirmed pending transactions after store snapshot manifest to db

* use log instead of fmt.Printf and remove unused var

* set current height to pending transaction that would be do apply unconfirmed

* 844 multisig changes (#854)

* #844 separate multisignature check to helper

* #844 refactor multisig
- update multisig rules
- move complicated database access to helper function

* #844 more test cases for multisig

* return array of pending signatures and pending transactions properly

* fix: checkMultisigComplete consider offchain multisig

* #844 index some multisig tables

* #844 test for helpers

* fix order of chain statuses in the returned response (#862)

fix order of chain statuses in the returned response (#862)

* Unit Test P2p handler p2pServerHandler.go (#863)

* GetMorePeers

* SendPeers

* GetCommonMilestoneBlockID

* GetNextBlocks

* GetPeerInfo

* GetCumulativeDifficulty

* GetNextBlockIDs

* SendBlock

* SendTransaction

* SendBlockTransactions

* RequestBlockTransactions

* Fix executor usages (#870)

* fix: handle duplicate rollback : level(panic)

* fix: handle rollback+commit() call in cmd.rollbackBlockchain : level(panic)

* fix: handle missing Executor.Rollback() call  : level(deadlock)

* fix: handle missing Executor.Rollback() call  : level(deadlock)

* fix: handle Commit() after Rollback()  : level(panic)

* CMD genesis generator and update circleci config (#865)

* cmd genesis generator and update circleci config

* fix golangci warning

* remove unused constant

* build job only for staging and develop

* account balance 0 as default

* wrong path given when get root via wd

* Dynamic genesis (#873)

* fixing typo CIRCLECI_BRANCH

* first commit (#866)

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

* Unit test p2p native strategy (#868)

* GetMorePeersHandler

* GetResolvedPeers

* GetAnyResolvedPeer

* AddToResolvedPeer

* RemoveResolvedPeer

* GetUnresolvedPeers

* AddToUnresolvedPeer

* AddToUnresolvedPeers

* RemoveUnresolvedPeer

* GetBlacklistedPeers

* AddToBlacklistPeer

* RemoveBlacklistedPeer

* GetAnyKnownPeer

* PeerBlacklist

* PeerUnblacklist

Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: Ariasa Ketut <[email protected]>

Co-authored-by: iltoga <[email protected]>
Co-authored-by: Andy Shi <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Gede Sukra Widhyawan <[email protected]>
Co-authored-by: Ahmad Ali Abdilah <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Co-authored-by: stefano galassi <[email protected]>
Co-authored-by: Kevin Hadi <[email protected]>
Co-authored-by: Nawi Kartini <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ENHANCE HIGH_PRIORITY needed ASAP for current sprint
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants