Admittedly the perplexity isn't at all good, but as per @jmtatsch that's likely due to llama's designed context of 2048:

$ pip list | grep llama
llama-cpp-python         0.1.57

$ python ./high_level_api_inference.py 
llama.cpp: loading model from /data/llama/7B/ggml-model-f16.bin
llama_model_load_internal: format     = ggjt v1 (pre #1405)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 8192
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 1 (mostly F16)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.07 MB
llama_model_load_internal: mem required  = 2292.09 MB (+ 1026.00 MB per state)
llama_model_load_internal: [cublas] offloading 32 layers to GPU
llama_model_load_internal: [cublas] offloading output layer to GPU
llama_model_load_internal: [cublas] total VRAM used: 12353 MB
...................................................................................................
llama_init_from_file: kv self size  = 4096.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 

llama_print_timings:        load time =  1845.23 ms
llama_print_timings:      sample time =  1255.23 ms /  2048 runs   (    0.61 ms per token)
llama_print_timings: prompt eval time = 10921.96 ms /  2254 tokens (    4.85 ms per token)
llama_print_timings:        eval time = 301437.04 ms /  2047 runs   (  147.26 ms per token)
llama_print_timings:       total time = 366120.17 ms
{
  "id": "cmpl-70bf6f1a-1aa8-401d-ab2f-e9302aa61cd8",
  "object": "text_completion",
  "created": 1686154862,
  "model": "/data/llama/7B/ggml-model-f16.bin",
  "choices": [
    {

"text": "In the beginning God created the heaven and the earth.\nAnd the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters.\nAnd God said, Let there be light: and there was light.\nAnd God saw the light, that it was good: and God divided the light from the darkness.\nAnd God called the light Day, and the darkness he called Night. And the evening and the morning were the first day.\nAnd God said, Let there be a firmament in the midst of the waters, and let it divide the waters from the waters.\nAnd God made the firmament, and divided the waters which were under the firmament from the waters which were above the firmament: and it was so.\nAnd God called the firmament Heaven. And the evening and the morning were the second day.\nAnd God said, Let the waters under the heaven be gathered together unto one place, and let the dry land appear: and it was so.\nAnd God called the dry land Earth; and the gathering together of the waters called he Seas: and God saw that it was good.\nAnd God said, Let the earth bring forth grass, the herb yielding seed, and the fruit tree yielding fruit after his kind, whose seed is in itself, upon the earth: and it was so.\nAnd the earth brought forth grass, and herb yielding seed after his kind, and the tree yielding fruit, whose seed was in itself, after his kind: and God saw that it was good.\nAnd the evening and the morning were the third day.\nAnd God said, Let there be lights in the firmament of the heaven to divide the day from the night; and let them be for signs, and for seasons, and for days, and years:\nAnd let them be for lights in the firmament of the heaven to give light upon the earth: and it was so.\nAnd God made two great lights; the greater light to rule the day, and the lesser light to rule the night: he made the stars also.\nAnd God set them in the firmament of the heaven to give light upon the earth,\nAnd to rule over the day and over the night, and to divide the light from the darkness: and God saw that it was good.\nAnd the evening and the morning were the fourth day.\nAnd God said, Let the waters bring forth abundantly the moving creature that hath life, and fowl that may fly above the earth in the open firmament of heaven.\nAnd God created great whales, and every living creature that moveth, which the waters brought forth abundantly, after their kind, and every winged fowl after his kind: and God saw that it was good.\nAnd God blessed them, saying, Be fruitful, and multiply, and fill the waters in the seas, and let fowl multiply in the earth.\nAnd the evening and the morning were the fifth day.\nAnd God said, Let the earth bring forth the living creature after his kind, cattle, and creeping thing, and beast of the earth after his kind: and it was so.\nAnd God made the beast of the earth after his kind, and cattle after their kind, and every thing that creepeth upon the earth after his kind: and God saw that it was good.\nAnd God said, Let us make man in our image, after our likeness: and let them have dominion over the fish of the sea, and over the fowl of the air, and over the cattle, and over all the earth, and over every creeping thing that creepeth upon the earth.\nSo God created man in his own image, in the image of God created he him; male and female created he them.\nAnd God blessed them, and God said unto them, Be fruitful, and multiply, and replenish the earth, and subdue it: and have dominion over the fish of the sea, and over the fowl of the air, and over every living thing that moveth upon the earth.\nAnd God said, Behold, I have given you every herb bearing seed, which is upon the face of all the earth, and every tree, in the which is the fruit of a tree yielding seed; to you it shall be for meat.\nAnd to every beast of the earth, and to every fowl of the air, and to every thing that creepeth upon the earth, wherein there is life, I have given every green herb for meat: and it was so.\nAnd God saw every thing that he had made, and, behold, it was very good. And the evening and the morning were the sixth day.\nGen.2\nThus the heavens and the earth were finished, and all the host of them.\nAnd on the seventh day God ended his work which he had made; and he rested on the seventh day from all his work which he had made.\nAnd God blessed the seventh day, and sanctified it: because that in it he had rested from all his work which God created and made.\nThese are the generations of the heavens and of the earth when they were created, in the day that the LORD God made the earth and the heavens,\nAnd every plant of the field before it was in the earth, and every herb of the field before it grew: for the LORD God had not caused it to rain upon the earth, and there was not a man to till the ground.\nBut there went up a mist from the earth, and watered the whole face of the ground.\nAnd the LORD God formed man of the dust of the ground, and breathed into his nostrils the breath of life; and man became a living soul.\nAnd the LORD God planted a garden eastward in Eden; and there he put the man whom he had formed.\nAnd out of the ground made the LORD God to grow every tree that is pleasant to the sight, and good for food; the tree of life also in the midst of the garden, and the tree of knowledge of good and evil.\nAnd a river went out of Eden to water the garden; and from thence it was parted, and became into four heads.\nThe name of the first is Pison: that is it which compasseth the whole land of Havilah, where there is gold;\nAnd the gold of that land is good: there is bdellium and the onyx stone.\nAnd the name of the second river is Gihon: the same is it that compasseth the whole land of Ethiopia.\nAnd the name of the third river is Hiddekel: that is it which goeth toward the east of Assyria. And the fourth river is Euphrates.\nAnd the LORD God took the man, and put him into the garden of Eden to dress it and to keep it.\nAnd the LORD God commanded the man, saying, Of every tree of the garden thou mayest freely eat:\nBut of the tree of the knowledge of good and evil, thou shalt not eat of it: for in the day that thou eatest thereof thou shalt surely die.\nAnd the LORD God said, It is not good that the man should be alone; I will make him an help meet for him.\nAnd out of the ground the LORD God formed every beast of the field, and every fowl of the air; and brought them unto Adam to see what he would call them: and whatsoever Adam called every living creature, that was the name thereof.\nAnd Adam gave names to all cattle, and to the fowl of the air, and to every beast of the field; but for Adam there was not found an help meet for him.\nAnd the LORD God caused a deep sleep to fall upon Adam and he slept: and he took one of his ribs, and closed up the flesh instead thereof;\nAnd the rib, which the LORD God had taken from man, made he a woman, and brought her unto the man.\nAnd Adam said, This is now bone of my bones, and flesh of my flesh: she shall be called Woman, because she was taken out of Man.\nTherefore shall a man leave his father and his mother, and shall cleave unto his wife: and they shall be one flesh.\nAnd they were both naked, the man and his wife, and were not ashamed.\nGen.3\nNow the serpent was more subtil than any beast of the field which the LORD God had made. And he said unto the woman, Yea, hath God said, Ye shall not eat of every tree of the garden?\nAnd the woman said unto the serpent, We may eat of the fruit of the trees of the garden:\nBut of the fruit of the tree which is in the midst of the garden, God hath said, Ye shall not eat of it, neither shall ye touch it, lest ye die.\nAnd the serpent said unto the woman, Ye shall not surely die:\nFor God doth know that in the day ye eat thereof, then your eyes shall be opened, and ye shall be as gods, knowing good and evil.\nAnd when the woman saw that the tree was good for food, and that it was pleasant to the eyes, and a tree to be desired to make one wise, she took of the fruit thereof, and did eat, and gave also unto her husband with her; and he did eat.\nAnd the eyes of them both were opened, and they knew that they were naked; and they sewed fig leaves together, and made themselves aprons.\nAnd they heard the voice of the LORD God walking in the garden in the cool of the day: and Adam and his wife hid themselves from the presence of the LORD God amongst the trees of the garden.\nAnd the LORD God called unto Adam, and said unto him, Where art thou?\nAnd he said, I heard thy voice in the garden, and I was afraid, because I was naked; and I hid myself.\nAnd he said, Who told thee that thou wast naked? Hast thou eaten of the tree, whereof I commanded thee that thou shouldest not eat?\nAnd the man said, The woman whom thou gavest to be with me, she gave me of the tree, and I did eat.\nAnd the man called unto him, and he ate: And the woman were two of the tree, and I shall be one.\nIn Adam, He was naked: And he ate:\nAnd the woman and man gave, and he ate,\nAnd it was dry land and the deep, and the man became dark.\nThe deep and the light and He called, the depth, and the darkness and the light, and the second day, the earth, the light, [the light, the darkness, and the light, the deep. And the light, and the dark, and The earth was the light: and the light. In the light, the and the light. the darkness and light, the night the the earth , the of the earth: the darkness and darkness made the the and the darkness, the and, the evening and the day and the earth\nAnd the Earth and darkness and the first and the light the day, and the darkness Day, and the night day and earth the day, and the day the and day the day, day and darkness Day, the, And the Day, the day, the day and and the day and the earth. The earth, and day there the 4th day there, dark day, the earth:\nHe the earth. Day the light in the day the day the day the day of the day and, the and, the. and earth, the earth, and the and it day, the day , the day and the day the day the and the earth, the earth ,, the earth -the, \u00c2 the, the darkness the, the there in the there is the was the was the, the was, Wearness,the was made there is, \u00c2 and the of, the waters, ..., the ground, the, the and the, and the earth the,\nIn [was the...in the unto, \u2026. The (In , (The the (cab -ch in the part the is the day, theness (the (of the, [. .. -the in, the Wether \u2013 of the [( (,[ of the, [...] of the*\t and (\n, [ ( (.\u00c2 [,\nThe,..., \u2022 of the, [W, ((Wails\nFewness ((Wather, ([,[((,, ,,, ([ is -(, ([, [[, [,. (., (. [. [\nof of the of the of the of. of the of the,[,, of the\u00c2 in it, [-the \u2014 [ (, \u2013, [\u2014(,, (,,.;, ([ (was, is (\u00ad[ (1[ of the of the of the of the ([\u0097 ( [ight of that of Wound\n[ (\ufffd [, ([,*, was,, in,, [,, ,.,, ([,, of the on,, the, as [\u200b, in,,, (> -\u00e2.\nM of the ,* ,-the of the of the of,, , , of the\u00c2 of the of the of of the of the [, [,, [,, ,, \u2014, [[,\u00c2, (W, was, is, of the, in of,,, of ( of [ of the of the [ of [ of the, of, of\nIn of the, of,, - of the of the of the of, of the,, (, of the, of ( of the,, of W of the of the, of , of of IN of, of the of of, [\u00ad, of, of [[ of the, ,,,, (, in of,, came,, ,,,, the,, (,,,, the, of the, of the, of the of the of, of, of the, of [, , of. of, of the of, of of of of of, of of of of (., of of, of of w of, of the of of of of of of of of of of of of of, . of of of of of water of of, --,,,. of, (,,,,,,,,, of [ ,,, (,, [,, [ ( [ ,,,,, of [ (, of [,,, [,, of of, of,,, ,, of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of, of of of of (, (,,, of [,,,,, [,,, (, [ of, ,, the, of of of of of of of of of of of of of of of of of of of, of of of of of of of of of the, ,,,,,,,,,, in of, ab,,,, prov,, [, of,,,, ( of of ( of [ [, of [ , of (\u00c2, [, [ [, [[ ,,, ab, of , of, of, of ,, d ,, of (, , of of the of a, of, of of of of of of of, of of of of of of of of of, of of of of of of of of of of of of of of [ of of of of of of of of of of of of the of of h of, (([ [ [(\u00ad, af [[, of of [, of [ [ ( [ [ [ [ [[ [ ( ( [ (, [ , the w [ [ ( \u2014 ( the ( [ of ( of of the of [[,. [{ [\u00ad [ ,, W. [, of , of of of of [\u00c2. [ of a ab, d over re ad, [,, ( ( ,,, [, [,, to- ., [ and,\t, c [ , [ [ \u2014, ( , ( , of, of of [ of the m. (\n or, of [ w de en [ , of of of of w of of [ of [ of of of h of of of [ of the of of of of of of of, . of of of of of of [ of of of [, [ [ and of of [ ( [, [, [. , [( . [. the [, [., [. [[ the the the the the d t,, .\u00c2 of the the [ [, of [ of of [\t of of of the of of of\u00ad of of of of of\n,. ( in the of, the of the ,. ay, \u2014 [ (, [ [, (, (,, [\u2009- . [, ,, , , ,, the of [ of , [\u00c2 ,,. , (,,,,,, [ [, [,.\u00ad, . [, the,, ,,,. the, the, . [ [ [ , of ( [\u00c2 of [ [, [, . of of of of of of of of of of of of of of of of of of of of of of of of of of of of, [,\n. a.\u2009. the [\u00ad [\u00ad( . ,, ,, d( the the the the the the [[ [[- [of the of of of of of of\u200b of [ of of . of of of of of of,, ( (\u00ad, the of, the , of the of , . of of af of of the the the the w,\u00c2 the. ( . [\u00c2\u00ad, (\u00c2 , - d , of w., al . (m ., ,, ,(\u00ad\u00c2, . . , . (, ,\u00ad . r , . ,. the , the the ., , , , [ ,, the [ [ [\u00ad in [ [, [ , . , \u2014. , [\u200b: , [ . ,{ ,, ,\u00ad ,,, , [, ,, of of of of of of [ ( . [( ( , , [, [. [, , [ [\u00c2 h to w, [, [\u00e2[[ , in (, [ [[ [\u00ad [? [ [\u009d [ ( [\u2009 , [ ( [ .. [ r, [ [., ,\u00c2 ( , [ [ , b . [ , , d [ , [ n, of , , . of of ( of of of , (\u00ad of of . , [\u00c2, , , [, , , of , , , , , , , ,\u200b\u00ad , ,[ , h [ [[ ab, , ( the, n, \u2014 ,, . ,: , ,, ., , and, [, , , [, -\u009d , , . , , in [ . [ [[ of [ of the the the the , . , [\ufffd the of the of [\u00ad of -- [( [\u00c2 of [\u00ad of . , . , , . . a to ,[ ,{ of \u201a of [\u0097\u00c2 of , ., [[ , ,, e [ , of\u00ad [ , [\u200d [ [\u00ad [\u00e2 [ [ (\u00c2 of n-of of of of . of of of of of of [ of of of of of, of [ of , . , [ of ,: of{\ufffd,\u00c2 . . . , . the\u00c2 [\u00ad , . ,\u00c2 \u00c2 [ , ,\n. [c, , , , , , [ . ,\u00c2 d ,",

      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 2254,
    "completion_tokens": 2048,
    "total_tokens": 4302
  }
}

high_level_api_inference.py.txt
genesis70.txt

[Updated issue] Prompt + Generation is limited to n_ctx. #331

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions