Enable TensorFlowTransform to work with pre-trained models that are not frozen #853

abgoswam · 2018-09-07T00:17:43Z

Fixes #784

Added ability in the TensorFlowTransform to to handle TensorFlow's SavedModel format
For doing this, we use a temporary directory for TF SavedModel files. It looks like this is a TensorFlow requirement. TensorFlow requires the model to be in a directory and doesn't provide a mechanism for loading from some sort of stream/memory representation.
The temp directory is created with appropriate ACL's for high-priviledge processes. We take care of deleting the temporary directory
Added unit tests.

yaeldekel · 2018-09-07T16:28:01Z

Should the model files go in the separate repo that we use for the other TensorFlow models? #Resolved

Ivanidzo4ka · 2018-09-07T19:58:23Z

// Licensed to the .NET Foundation under one or more agreements.

Can you merge it with master?
I've just push changes in this file to introduce estimator and transformer objects #Resolved

Refers to: src/Microsoft.ML.TensorFlow/TensorflowTransform.cs:1 in 57508d3. [](commit_id = 57508d3, deletion_comment = False)

Ivanidzo4ka · 2018-09-07T19:58:55Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

            }

            public void Save(ModelSaveContext ctx)
            {
                _host.AssertValue(ctx);
                ctx.CheckAtModel();
                ctx.SetVersionInfo(GetVersionInfo());
+                ctx.Writer.Write(_isFrozen ? 1 : 0);


Write [](start = 27, length = 5)

I believe you can write bool #Resolved

ctx.Writer.WriteBoolByte(IsFrozen);

In reply to: 216071878 [](ancestors = 216071878)

Ivanidzo4ka · 2018-09-07T20:00:32Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

            }

            public void Save(ModelSaveContext ctx)
            {
                _host.AssertValue(ctx);
                ctx.CheckAtModel();
                ctx.SetVersionInfo(GetVersionInfo());
+                ctx.Writer.Write(_isFrozen ? 1 : 0);
+                if (_isFrozen)


_isFrozen [](start = 20, length = 9)

Please remain saving order, so you can read old version of model files. #Resolved

Ivanidzo4ka · 2018-09-07T20:02:16Z

test/Microsoft.ML.Core.Tests/UnitTests/TestEntryPoints.cs

@@ -235,6 +235,8 @@ private string GetBuildPrefix()
        }

        [Fact(Skip = "Execute this test if you want to regenerate ep-list and _manifest.json")]
+        // [Fact]


// [Fact] [](start = 7, length = 10)

not necessary #Resolved

…machinelearning into abgoswam/tf_savedmodel

Ivanidzo4ka · 2018-09-10T23:31:05Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

            public string[] OutputColumns;
        }

        private readonly IHost _host;
        private const string RegistrationName = "TensorFlowTransform";

        internal readonly TFSession Session;
+        internal readonly bool IsFrozen;


IsFrozen [](start = 31, length = 8)

You don't need them. All you care about is session, and you already have it. This is just details of how session been loaded. #Resolved

…dels

…ML.TensorFlow.TestModels contains these models

abgoswam · 2018-09-12T17:33:50Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+                return new TensorFlowTransform(env, modelBytes, io.Item1, io.Item2, (isFrozen == 1));
+            }
+            else
+            {


{ [](start = 11, length = 2)

[observation] The Train() method of the LearningPipeline API ends up calling Save() and Create() multiple times.

For un-frozen models we need a particular directory structure so we can create the TFSession correctly using the TFSession.FromSavedModel API (line 215)

In this call to Create() we do the following :
(a) load the binary stream
(b) write the binary stream out to a zip file
(c) extract the zip file to a temporary folder TempSavedModelDirName (_savedModel)
(d) create the session
(e) delete the zip file

We do not delete the temporary folder because in a subsequent call to Save() we are writing out the contents of the temporary folder as a byte stream.

So for this scenario, what we have currently is that we define a unique location for the temporary folder -- and it gets cleaned up only when required, inside the Create() call. Line 139.

Is there some way to ensure the temporary folder is always cleaned up ? Or perhaps some way to not have a temporary folder at all.

@ericstj @zeahmed @yaeldekel #Resolved

You should avoid the temporary zip file. If you can get a seekable stream from ctx, just pass that into the ZipArchive constructor: https://msdn.microsoft.com/es-es/library/system.io.compression.ziparchive.ziparchive(v=vs.110).aspx. It looks like you can do this by accessing the BaseStream on the binary reader passed into the TryLoadBinaryStream action.

You should also avoid allocating an extra byte array for the zip and instead do all the reading within the callback from TryLoadBinaryStream, since it looks like TryLoadBinaryStream will close the stream:

machinelearning/src/Microsoft.ML.Data/Model/ModelLoading.cs

Line 292 in 0d5e317

using (ent)

. So within this callback, new up the archive, then call archive.ExtractToDirectory extension method: https://docs.microsoft.com/en-us/dotnet/api/system.io.compression.zipfileextensions.extracttodirectory?view=netframework-4.7.2.

I think you still need to have a temporary directory for the model files themselves. It looks like this is a requirement of TensorFlow https://github.com/tensorflow/tensorflow/blob/3be04971716fcaf0c11ad9262e60efa428553e14/tensorflow/c/c_api.h#L1367. It requires the model to be in a directory and doesn't provide a mechanism for loading from some sort of stream/memory representation. So you'll still need a temp path that you'll have to clean up.

Ideally you should provide a way for this path to be specified, rather than always call GetTempPath behind the scenes. I think ML.NET has a pattern for this, if not, perhaps consider a constructor overload that allows for specification of a temp path root. You may also consider some way to say "don't clean up the temp path".

Typically with things like this you should treat the temporary folder as state managed by the object and then cleaned up in the dispose/finalizer.

In reply to: 217123233 [](ancestors = 217123233)

I tried both of the suggestions above, but they seem to have some limitations with respect to this transform. I have since modified the logic of load&restore to do things in-memory.

Am documenting the issues hit with the above suggestion:

(1) the ZipArchive approach looked promising to avoid the temporary zip file. However it works on the entire stream, rather than on a subset of the stream.
(2) currently the TFTransform does not dispose the TFSession object. So attempts to cleanup the temp folder fail with 'file still in use' errors. The approach of using unique location for the temporary folder is problematic in general (e.g. during CV or if pipeline contains multiple TFTransforms)

I have modified the logic so we do the ML.NET serialization/de-serialization using in-memory data structures.

In reply to: 217145506 [](ancestors = 217145506,217123233)

You should file an issue on #2.

I think this approach looks much better than zip-in-a-zip approach before. One change I'd suggest is that instead of reading to byte array and storing in memory, you create the file within the callback (var fs = new FileStream(...)) and use br.BaseStream.CopyTo(fs, fileLength); To do this you'll need to replace readbytearray/writebytearray with a your own calls that store file length, followed by the stream copy.

One thing I notice that could be problematic for the frozen case above (not related to this change) is that it seems to use Graph.Import to import a entire model as a managed byte array. I can imagine that this will potentially hit the upper limit of managed array sizes for large models. I see that TF's graph import functionality expects a single buffer (https://github.com/tensorflow/tensorflow/blob/3be04971716fcaf0c11ad9262e60efa428553e14/tensorflow/c/c_api.h#L1018-L1020).

In reply to: 217447210 [](ancestors = 217447210,217145506,217123233)

The main reason for reading to byte array and storing in memory is to handle the scenario where we saw the sequence of Save1() -> Create1() -> Save2() -> Create2() being called

When Create1() is called, we load up the contents into a dictionary. So when Save2() is called we use the dictionary to write out the contents.

Because of issue #906 (not being able to clean-up unmanaged resources ) I am not sure we can do away with the in-memory approach. Thoughts ?

In reply to: 217462971 [](ancestors = 217462971,217447210,217145506,217123233)

adopted the stream copy based approach instead of reading to byte array

also added a finalizer that closes the session (if it isn’t closed) and deletes the temporary directory

In reply to: 217545311 [](ancestors = 217545311,217462971,217447210,217145506,217123233)

@eerhardt , @danmosemsft , would this be a security risk? To load the TensorFlow model here, we create a temporary directory and copy some files to it, and then load the model from that directory. #Resolved

followed the recommendation of the .NET security team to address this risk

In reply to: 218159541 [](ancestors = 218159541)

zeahmed · 2018-09-12T18:04:15Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

        private const string RegistrationName = "TensorFlowTransform";
+        private const string TempSavedModelDirName = "MLNET_TensorFlowTransform_SavedModel";
+        private const string TempSavedModelZipName = "SavedModel.zip";


private const string TempSavedModelZipName = "SavedModel.zip"; [](start = 8, length = 62)

What will happen if I am working on two different pipelines with TensorflowTransforms executing in parallel? I think having same name for .zip (SavedModel.zip) and model_dir (MLNET_TensorFlowTransform_SavedModel)will create a problem.

Let see what does @eric suggest. However, when I did initially, my approach was as follows

In the constructor, load all the files in un-frozen model into a dictionary (called model dictionary) where keys are the file name and value is the stream of file.

Load the model from the location that user provided.

When saving, save the streams in the model dictionary into ML.Net stream.

When loading from ML.Net stream, load the streams into the model dictionary, create temporary folder and write all the stream into the folder. Load the model into TF using that temporary folder. Delete the model once loading is done.

#Resolved

This obviously has a limitation on the size of model.

In reply to: 217133381 [](ancestors = 217133381)

True. I have modified the logic so we do the load&restore in-memory (using a dictionary).

In reply to: 217134254 [](ancestors = 217134254,217133381)

yaeldekel · 2018-09-20T19:35:35Z

src/Microsoft.ML.TensorFlow/TensorFlow/TensorflowUtils.cs

+                subDirInfo.Delete(true);
+            }
+        }
+        internal static TFSession GetSession(IHostEnvironment env, string modelPath)


internal [](start = 8, length = 8)

nit: Add an empty line between methods. #Closed

yaeldekel · 2018-09-20T19:51:41Z

    public void Dispose()

@TomFinley or @Zruty0 might be able to give more insight. I think that with the Estimator/Transformer API it might cause problems: The TensorFlowEstimator instantiates a TensorFlowTransform in its constructor, which is returned to the user when Fit() is called. The estimator still owns the TensorFlowTransform though, and it will return the same one if Fit() is called again, so we wouldn't want someone to dispose it.

In reply to: 423241990 [](ancestors = 423241990,423236224)

Refers to: src/Microsoft.ML.TensorFlow/TensorflowTransform.cs:401 in ae672d6. [](commit_id = ae672d6, deletion_comment = True)

ericstj · 2018-09-20T20:22:53Z

    public void Dispose()

Is the Estimator the better place for this state to live? At the end of the day, the directory is representing the "training state" and it needs to be managed by some object. That object should be responsible for cleaning it up and should have a definite lifetime.

In reply to: 423310984 [](ancestors = 423310984,423241990,423236224)

Refers to: src/Microsoft.ML.TensorFlow/TensorflowTransform.cs:401 in ae672d6. [](commit_id = ae672d6, deletion_comment = True)

Ivanidzo4ka · 2018-09-21T17:31:52Z

src/Microsoft.ML.TensorFlow/Microsoft.ML.TensorFlow.csproj

+  <ItemGroup>
+    <PackageReference Include="System.IO.FileSystem.AccessControl" Version="4.5.0" />
+    <PackageReference Include="System.Security.Principal.Windows" Version="4.5.0" />
+  </ItemGroup>


A) Do we really need them? Because presence of Principal.Windows, for multiplatform code is suspicious.
B) If we really need them you need to make them part of nuget dependency, and also we need to modify https://github.com/dotnet/machinelearning/blob/master/build/Dependencies.props instead of specify versions here. #Resolved

We use this as a security measure on windows platform. We are using temp directory for loading TFSession for SavedModels. Models are considered executable code, so for this temp directory we need to ACL it in the high-rights process so low-rights process can’t access it.

I will look into making it part of nuget dependency

In reply to: 219572600 [](ancestors = 219572600)

you can use this one as reference implementation https://github.com/dotnet/machinelearning/blob/master/pkg/Microsoft.ML.ImageAnalytics/Microsoft.ML.ImageAnalytics.nupkgproj

In reply to: 219577320 [](ancestors = 219577320,219572600)

Ivanidzo4ka · 2018-09-21T17:42:38Z

src/Microsoft.ML.TensorFlow/TensorFlow/TensorflowUtils.cs

+            return CheckFileAndRead(env, modelPath);
+        }
+
+        private static TFSession CheckFileAndRead(IHostEnvironment env, string modelFile)


private static TFSession CheckFileAndRead(IHostEnvironment env, string modelFile) [](start = 8, length = 81)

can you get rid of this method? no one using it any more except GetSession, and I think it would be much better to move checks in start of GetSesssion #Resolved

Ivanidzo4ka · 2018-09-21T17:47:59Z

src/Microsoft.ML.TensorFlow/TensorFlow/TensorflowUtils.cs

+            return attr.HasFlag(FileAttributes.Directory);
+        }
+
+        internal static void CreateTempDirectory(string tempDirPath)


CreateTempDirectory [](start = 29, length = 19)

What's the main reason for this? Why we treat Windows differently?
If this is necessary, why it's part of tensorflow only and not part of utility which we use among whole project? #Resolved

(replied on other comment about its necessity)

Currently used in TFTransform only to protect SavedModels which are considered executable code by .NET security team

In reply to: 219577130 [](ancestors = 219577130)

Can you put comment with description why we doing this?
I would be really tempted to remove this code if I wouldn't know story behind it.

In reply to: 219578542 [](ancestors = 219578542,219577130)

…th vector

abgoswam · 2018-09-21T23:04:12Z

    public void Dispose()

As suggested marking as Pending for now. We do not want block this PR on this. Will track it as part of issue #906

In reply to: 423320325 [](ancestors = 423320325,423310984,423241990,423236224)

Refers to: src/Microsoft.ML.TensorFlow/TensorflowTransform.cs:401 in ae672d6. [](commit_id = ae672d6, deletion_comment = True)

zeahmed · 2018-09-24T16:33:15Z

src/Microsoft.ML.TensorFlow/TensorFlow/TensorflowUtils.cs

+        ///  Given a folder path, create it with proper ACL if it doesn't exist.
+        ///  Fails if the folder name is empty, or can't create the folder.
+        /// </summary>
+        internal static void CreateFolderWithAclIfNotExists(IHostEnvironment env, string folder)


CreateFolderWithAclIfNotExists [](start = 29, length = 30)

How does creation of ACL folder work in Linux/Mac? #Closed

Catches the PlatformNotSupported exception and does the normal create.

In reply to: 219903672 [](ancestors = 219903672)

Thanks Eric, got it.

In reply to: 219906478 [](ancestors = 219906478,219903672)

…eleting temp folder (3) deleted test using Legacy Learning API

zeahmed

Ivanidzo4ka · 2018-09-25T00:25:26Z

pkg/Microsoft.ML.TensorFlow/Microsoft.ML.TensorFlow.nupkgproj

+	<PackageReference Include="System.IO.FileSystem.AccessControl" Version="$(SystemIOFileSystemAccessControl)" />
+	<PackageReference Include="System.Security.Principal.Windows" Version="$(SystemSecurityPrincipalWindows)" />
+	<ProjectReference Include="../Microsoft.ML/Microsoft.ML.nupkgproj" />    
+	<ProjectReference Include="../Microsoft.ML.TensorFlow.Redist/Microsoft.ML.TensorFlow.Redist.nupkgproj" />


minor, but please use spaces #Resolved

yaeldekel

Ivanidzo4ka · 2018-09-25T00:44:37Z

src/Microsoft.ML.TensorFlow/TensorFlow/TensorflowUtils.cs

+        }
+
+        internal static TFSession GetSession(IHostEnvironment env, string modelPath)
+        {


Contracts.Check(env, nameof(env); #Resolved

added this check.

btw i believe you meant Contracts.Check(env != null, nameof(env));

In reply to: 220031028 [](ancestors = 220031028)

Ivanidzo4ka · 2018-09-25T01:05:53Z

src/Microsoft.ML.TensorFlow/TensorFlow/TensorflowUtils.cs

+        {
+            env.CheckValue(exportDirSavedModel, nameof(exportDirSavedModel));
+            var sessionOptions = new TFSessionOptions();
+            var exportDir = exportDirSavedModel;


var exportDir = exportDirSavedModel; [](start = 11, length = 37)

not needed. #Resolved

Ivanidzo4ka

…t of Microsoft.ML.TensorFlow

eerhardt · 2018-09-25T19:45:07Z

src/Microsoft.ML.DnnAnalyzer/Microsoft.ML.DnnAnalyzer/Microsoft.ML.DnnAnalyzer.csproj

@@ -2,7 +2,7 @@

  <PropertyGroup>
    <OutputType>Exe</OutputType>
-    <TargetFramework>netcoreapp2.1</TargetFramework>
+    <TargetFramework>netcoreapp2.0</TargetFramework>


This should be netcoreapp2.1. Note that netcoreapp and netstandard are different things. We want netcoreapp2.1 for any and all executables and tests. We want netstandard2.0 for any libraries.

Moreover, we don't even want this in the lib folder as it is just a commandline tool.

abgoswam added 5 commits September 6, 2018 01:04

building transform from ground up

1007955

dummy transform works after fixing the getters

40fbedc

SavedModel format works for Train, but fails for Save&Predict

48d14c6

remove dummy transform

35ff43a

remove dummy unit test

6291d0d

abgoswam added 3 commits September 7, 2018 19:13

Works with non-frozen models

57508d3

building transform from ground up

cfcd70f

dummy transform works after fixing the getters

236de73

Ivanidzo4ka reviewed Sep 7, 2018

View reviewed changes

abgoswam added 9 commits September 7, 2018 20:06

SavedModel format works for Train, but fails for Save&Predict

781cff0

remove dummy transform

07b15a0

remove dummy unit test

47f75b5

Merge branch 'abgoswam/tf_savedmodel' of https://github.com/abgoswam/…

c304257

…machinelearning into abgoswam/tf_savedmodel

merge with master

d0430b5

fix compilation issues; verify existing tests work fine

950a210

works locally; need to refactor code

97eb497

refactored code; keeping only 1 version of the convenience API

173729f

Merge branch 'master' into abgoswam/tf_savedmodel

292140b

Ivanidzo4ka reviewed Sep 10, 2018

View reviewed changes

abgoswam added 5 commits September 11, 2018 16:07

added class for directory structure

655a8aa

using latest nuget package (0.0.3) for Microsoft.ML.TensorFlow.TestMo…

be5285a

…dels

delete temporary files used when loading/saving models

46c04a3

delete local models; the updated nuget version (0.0.3) for Microsoft.…

e705f93

…ML.TensorFlow.TestModels contains these models

modified logic for load/restore of models

84214e2

abgoswam commented Sep 12, 2018

View reviewed changes

zeahmed reviewed Sep 12, 2018

View reviewed changes

yaeldekel reviewed Sep 20, 2018

View reviewed changes

merge with latest master

21879f6

Ivanidzo4ka reviewed Sep 21, 2018

View reviewed changes

abgoswam added 2 commits September 21, 2018 22:48

fix issues with nuget packaging; refactored the code + added comments

a1d912d

add checks in code to make sure that the input is not a variable leng…

f6a1c84

…th vector

abgoswam mentioned this pull request Sep 21, 2018

TensorFlowTransform does not implement IDisposable. #906

Closed

abgoswam added 2 commits September 22, 2018 03:33

merge with latest master

5957c53

fix typo in name of package

5120bb9

zeahmed reviewed Sep 24, 2018

View reviewed changes

abgoswam added 2 commits September 24, 2018 18:50

(1) added SavedModel test for MNIST model (2) added try/finally for d…

a624a3b

…eleting temp folder (3) deleted test using Legacy Learning API

remove and sort usings in file TrainSaveModelAndPredict.cs

8d9fdc5

zeahmed approved these changes Sep 24, 2018

View reviewed changes

Ivanidzo4ka reviewed Sep 25, 2018

View reviewed changes

using spaces in nupkgproj

685ab99

yaeldekel approved these changes Sep 25, 2018

View reviewed changes

Ivanidzo4ka reviewed Sep 25, 2018

View reviewed changes

Ivanidzo4ka approved these changes Sep 25, 2018

View reviewed changes

abgoswam added 2 commits September 25, 2018 16:25

error checking for passed in IHostEnvironment

2d0ec1e

fix TargetFramework version (netcore 2.0) of DnnAnalyzer to match tha…

8d8b986

…t of Microsoft.ML.TensorFlow

abgoswam merged commit 893a385 into dotnet:master Sep 25, 2018

abgoswam mentioned this pull request Sep 25, 2018

Use TextLoader.Create in unit test TensorFlowTransformCifarSavedModel #1025

Merged

eerhardt reviewed Sep 25, 2018

View reviewed changes

abgoswam deleted the abgoswam/tf_savedmodel branch November 2, 2018 21:07

ghost locked as resolved and limited conversation to collaborators Mar 29, 2022

Enable TensorFlowTransform to work with pre-trained models that are not frozen #853

Enable TensorFlowTransform to work with pre-trained models that are not frozen #853

Conversation

abgoswam commented Sep 7, 2018 • edited Loading

yaeldekel commented Sep 7, 2018 • edited by abgoswam Loading

Ivanidzo4ka commented Sep 7, 2018 • edited by abgoswam Loading

Ivanidzo4ka Sep 7, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 7, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 7, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 10, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

abgoswam Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

ericstj Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericstj Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

abgoswam Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaeldekel Sep 17, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeahmed Sep 12, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

zeahmed Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaeldekel Sep 20, 2018 • edited Loading

Choose a reason for hiding this comment

yaeldekel commented Sep 20, 2018

ericstj commented Sep 20, 2018

Ivanidzo4ka Sep 21, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

abgoswam Sep 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 21, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 21, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

abgoswam Sep 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam commented Sep 21, 2018 • edited Loading

zeahmed Sep 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeahmed Sep 24, 2018 • edited Loading

Choose a reason for hiding this comment

zeahmed left a comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 25, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

yaeldekel left a comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 25, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 25, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam commented Sep 7, 2018 •

edited

Loading

yaeldekel commented Sep 7, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka commented Sep 7, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 7, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 7, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 7, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 10, 2018 •

edited by abgoswam

Loading

abgoswam Sep 12, 2018 •

edited

Loading

ericstj Sep 12, 2018 •

edited

Loading

ericstj Sep 13, 2018 •

edited

Loading

abgoswam Sep 13, 2018 •

edited

Loading

yaeldekel Sep 17, 2018 •

edited by abgoswam

Loading

zeahmed Sep 12, 2018 •

edited by abgoswam

Loading

zeahmed Sep 12, 2018 •

edited

Loading

yaeldekel Sep 20, 2018 •

edited

Loading

Ivanidzo4ka Sep 21, 2018 •

edited by abgoswam

Loading

abgoswam Sep 21, 2018 •

edited

Loading

Ivanidzo4ka Sep 21, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 21, 2018 •

edited by abgoswam

Loading

abgoswam Sep 21, 2018 •

edited

Loading

abgoswam commented Sep 21, 2018 •

edited

Loading

zeahmed Sep 24, 2018 •

edited

Loading

zeahmed Sep 24, 2018 •

edited

Loading

Ivanidzo4ka Sep 25, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 25, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Sep 25, 2018 •

edited by abgoswam

Loading