Fixing inconsistency in usage of LossFunction #2856

ganik · 2019-03-05T20:05:29Z

fixes #2174
fixes #2594

Merge

…ssFactory interfaces

Merge

Ivanidzo4ka · 2019-03-05T21:25:29Z

public interface IClassificationLoss : IScalarOutputLoss

Can we hide this one? #ByDesign

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:43 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

Ivanidzo4ka · 2019-03-05T21:25:44Z

public interface IRegressionLoss : IScalarOutputLoss

Can we hide this one? #ByDesign

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:33 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

Ivanidzo4ka · 2019-03-05T21:26:05Z

public sealed class LogLossFactory : ISupportSdcaClassificationLossFactory, ISupportClassificationLossFactory

Can it be internal? #Closed

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:98 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

ganik · 2019-03-05T22:04:35Z

public interface IClassificationLoss : IScalarOutputLoss

nope this cant as we have this one that needs to stay public: public interface ISupportSdcaClassificationLoss : ISupportSdcaLoss, IClassificationLoss

In reply to: 469863324 [](ancestors = 469863324)

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:43 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

ganik · 2019-03-05T22:04:45Z

public sealed class LogLossFactory : ISupportSdcaClassificationLossFactory, ISupportClassificationLossFactory

yep

In reply to: 469863522 [](ancestors = 469863522)

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:98 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

ganik · 2019-03-05T22:04:54Z

public interface IRegressionLoss : IScalarOutputLoss

nope this cant as we have this one that needs to stay public: public interface ISupportSdcaClassificationLoss : ISupportSdcaLoss, IClassificationLoss

In reply to: 469863409 [](ancestors = 469863409)

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:33 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

wschin · 2019-03-05T22:31:24Z

public interface IScalarOutputLoss : ILossFunction<float, float>

Mabye change it to IScalarLoss? The definition of Output is ambigous. #Resolved

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:19 in 21ea513. [](commit_id = 21ea513, deletion_comment = False)

wschin · 2019-03-05T22:32:24Z

    Double Loss(TOutput output, TLabel label);

Maybe we should sync its returned type with float Derivative(float output, float label); #WontFix

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:16 in 21ea513. [](commit_id = 21ea513, deletion_comment = False)

wschin · 2019-03-05T22:35:05Z

    Double DualLoss(float label, Double dual);

float maybe? #Resolved

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:72 in 21ea513. [](commit_id = 21ea513, deletion_comment = False)

wschin · 2019-03-05T22:36:14Z

src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs

+            internal ISupportSdcaClassificationLossFactory LossFunctionFactory = new LogLossFactory();
+
+            /// <summary>
+            /// The custom <a href="tmpurl_loss">loss</a>.


tmpurl_loss [](start = 36, length = 11)

What dose it link to? #Resolved

Shahab is working on this, issue #2356

In reply to: 262720628 [](ancestors = 262720628)

wschin · 2019-03-05T22:39:36Z

src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs

+            /// <value>
+            /// If unspecified, <see cref="LogLoss"/> will be used.
+            /// </value>
+            public ISupportSdcaClassificationLoss LossFunction;


LossFunction [](start = 50, length = 12)

It doesn't look like we're using the same pattern Tom mentioned this morning. Please take a look at the following example for hiding EarlyStoppingRuleFactory via adding a new EarlyStoppingRule.

/// <summary> /// Early stopping rule. (Validation set (/valid) is required). /// </summary> [BestFriend] [Argument(ArgumentType.Multiple, HelpText = "Early stopping rule. (Validation set (/valid) is required.)", ShortName = "esr", NullName = "<Disable>")] [TGUI(Label = "Early Stopping Rule", Description = "Early stopping rule. (Validation set (/valid) is required.)")] internal IEarlyStoppingCriterionFactory EarlyStoppingRuleFactory; private EarlyStoppingRuleBase _earlyStoppingRuleBase; public EarlyStoppingRuleBase EarlyStoppingRule { get { return _earlyStoppingRuleBase; } set { _earlyStoppingRuleBase = value; EarlyStoppingRuleFactory = _earlyStoppingRuleBase.BuildFactory(); } } ``` #Resolved

I dont see benefits of the changes you are proposing. What does _earlyStoppingRuleBase.BuildFactory(); do? How does it know what factory to build out if I set non default EarlyStoppingRule ?

In reply to: 262721736 [](ancestors = 262721736)

One benefit is that the way Wei-Sheng proposes means that the object does not really enter any sort of inconsistent state, even for our internal code, thereby making the object easier to maintain. We could of course say, since the only thing touching this object at that level is this code, that we can avoid any flaws, but -- let's be honest with ourselves, even internal inconsistencies often show up as bugs externally. :) If you don't want to do it right now @ganik that is fine with me, but I would prefer in that case if we could make it a property so that we could adjust it later, if we believe it is necessary, in subsequent versions, without it being a breaking change. (So we'd change, here, public ISupportSdcaClassificationLoss LossFunction { get; set; } so that it becomes an auto-implemented property.

In reply to: 262729182 [](ancestors = 262729182,262721736)

Just going to re-activate for tracking. Like I say, you don't have to do it right now, but I'd rather it be a property in this case.

In reply to: 263112293 [](ancestors = 263112293,262729182,262721736)

Sure I am fine with properties. For me the importance was that as third party dev I can supply my own LossFunction definition and will not be restricted to few pre-defined loss functions: LogLoss, HingeLoss etc. With Wei-Sheng pattern this becomes prohibitive.

In reply to: 263112433 [](ancestors = 263112433,263112293,262729182,262721736)

Modified to properties

In reply to: 263631467 [](ancestors = 263631467,263112433,263112293,262729182,262721736)

wschin · 2019-03-05T22:46:26Z

src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs

@@ -1666,7 +1680,7 @@ public sealed class Options : BinaryOptionsBase
        }

        internal SdcaNonCalibratedBinaryTrainer(IHostEnvironment env, Options options)
-            : base(env, options, options.LossFunction.CreateComponent(env))
+            : base(env, options, options.LossFunction ?? options.LossFunctionFactory.CreateComponent(env))


The only constructor of Options assigns new LogLoss() to LossFunction. Is it still possible to execute CreateComponent? #ByDesign

not anymore, LossFunction is null by default

In reply to: 262723959 [](ancestors = 262723959)

Then, user will see a trainer with null loss, but that trainer is minimizing LogLoss. That's why I implemented a pattern mentioned by Tom.

In reply to: 262724703 [](ancestors = 262724703,262723959)

sure, as per documentation line 1659:
If unspecified, will be used.

In reply to: 262726971 [](ancestors = 262726971,262724703,262723959)

I might prefer that the value be present (the idea of null acting like a particular specific instance is a bit suboptimal), but I don't insist on it I guess...

In reply to: 262728446 [](ancestors = 262728446,262726971,262724703,262723959)

wschin · 2019-03-05T22:47:24Z

src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs

+            /// </value>
+            public ISupportSdcaClassificationLoss LossFunction;
+
+            public Options()


Please add user-facing document. Thanks. #Resolved

constructor removed

In reply to: 262724236 [](ancestors = 262724236)

wschin · 2019-03-05T22:48:11Z

src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs

+            [Argument(ArgumentType.Multiple, Name = "LossFunction", HelpText = "Loss Function", ShortName = "loss", SortOrder = 50)]
+            public ISupportSdcaClassificationLossFactory LossFunctionFactory = new LogLossFactory();
+
+            public ISupportSdcaClassificationLoss LossFunction;


Please add user-facing document. #Resolved

changes are removed for this class

In reply to: 262724466 [](ancestors = 262724466)

wschin · 2019-03-05T22:48:20Z

src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs


            [Argument(ArgumentType.AtMostOnce, HelpText = "The calibrator kind to apply to the predictor. Specify null for no calibration", Visibility = ArgumentAttribute.VisibilityType.EntryPointsOnly)]
            internal ICalibratorTrainerFactory Calibrator = new PlattCalibratorTrainerFactory();

            [Argument(ArgumentType.AtMostOnce, HelpText = "The maximum number of examples to use when training the calibrator", Visibility = ArgumentAttribute.VisibilityType.EntryPointsOnly)]
            public int MaxCalibrationExamples = 1000000;
+
+            public Options()


Please add user-facing document. #Resolved

this is removed

In reply to: 262724506 [](ancestors = 262724506)

wschin · 2019-03-05T22:49:18Z

src/Microsoft.ML.StandardLearners/Standard/SdcaRegression.cs

+            internal ISupportSdcaRegressionLossFactory LossFunctionFactory = new SquaredLossFactory();
+
+            /// <summary>
+            /// A custom <a href="tmpurl_loss">loss</a>.


tmpurl_loss [](start = 34, length = 11)

Not sure what this is.. #Resolved

see the line 41? same documentation as for the loss factory

In reply to: 262724768 [](ancestors = 262724768)

Then please fix both. Maybe you can git blame to find the killer.

In reply to: 262725105 [](ancestors = 262725105,262724768)

Shahab is working on this, issue #2356

In reply to: 262726212 [](ancestors = 262726212,262725105,262724768)

wschin · 2019-03-05T22:51:49Z

Not sure if you need to touch this guy. It's internal. #Resolved

Refers to: src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs:1721 in 21ea513. [](commit_id = 21ea513, deletion_comment = False)

ganik · 2019-03-05T22:52:44Z

good point

In reply to: 469891311 [](ancestors = 469891311)

Refers to: src/Microsoft.ML.StandardLearners/Standard/SdcaBinary.cs:1721 in 21ea513. [](commit_id = 21ea513, deletion_comment = False)

ganik · 2019-03-05T23:33:27Z

public interface IScalarOutputLoss : ILossFunction<float, float>

yep i like that

In reply to: 469884938 [](ancestors = 469884938)

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:19 in 21ea513. [](commit_id = 21ea513, deletion_comment = False)

codecov · 2019-03-06T00:23:50Z

Codecov Report

Merging #2856 into master will increase coverage by 0.12%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master    #2856      +/-   ##
==========================================
+ Coverage   71.69%   71.81%   +0.12%     
==========================================
  Files         809      812       +3     
  Lines      142427   142643     +216     
  Branches    16112    16095      -17     
==========================================
+ Hits       102108   102437     +329     
+ Misses      35890    35824      -66     
+ Partials     4429     4382      -47

Flag	Coverage Δ
#Debug	`71.81% <85.71%> (+0.12%)`	⬆️
#production	`67.95% <85.71%> (+0.04%)`	⬆️
#test	`86.25% <ø> (+0.34%)`	⬆️

Impacted Files	Coverage Δ
test/Microsoft.ML.Core.Tests/UnitTests/TestLoss.cs	`100% <ø> (ø)`	⬆️
...StandardLearners/Standard/Online/AveragedLinear.cs	`59.5% <ø> (ø)`	⬆️
...dLearners/Standard/Online/OnlineGradientDescent.cs	`92.3% <100%> (+1%)`	⬆️
...crosoft.ML.StandardLearners/Standard/SdcaBinary.cs	`72.55% <100%> (+0.44%)`	⬆️
...oft.ML.StandardLearners/Standard/SdcaRegression.cs	`95.83% <100%> (-0.22%)`	⬇️
...oft.ML.StandardLearners/Standard/SdcaMultiClass.cs	`92.25% <100%> (+0.02%)`	⬆️
...dardLearners/Standard/Online/AveragedPerceptron.cs	`85.71% <100%> (+0.52%)`	⬆️
src/Microsoft.ML.Data/Utils/LossFunctions.cs	`68.39% <62.5%> (-1.45%)`	⬇️
src/Microsoft.ML.FastTree/RegressionTree.cs	`71.42% <0%> (-12.25%)`	⬇️
...soft.ML.FastTree/Training/EarlyStoppingCriteria.cs	`71.73% <0%> (-11.47%)`	⬇️
... and 547 more

TomFinley · 2019-03-06T19:53:12Z

public interface IClassificationLoss : IScalarOutputLoss

Note that the scope of this work is not to hide the loss functions -- as Gani points out that is impossible. THe scope of the work is to hide the empty-loss function factory interfaces, which exist merely to support the presence of this TlcModule.ComponentKind attribute.

In reply to: 469876577 [](ancestors = 469876577,469863324)

Refers to: src/Microsoft.ML.Data/Dirty/ILoss.cs:43 in 24c52e1. [](commit_id = 24c52e1, deletion_comment = False)

TomFinley · 2019-03-06T19:54:42Z

src/Microsoft.ML.StandardLearners/Standard/Online/AveragedPerceptron.cs

+            internal ISupportClassificationLossFactory LossFunctionFactory1 = new HingeLoss.Options();
+
+            /// <summary>
+            /// A custom <a href="tmpurl_loss">loss</a>.


tmpurl_loss [](start = 34, length = 11)

Were you intending to update this? #Resolved

Also I noted elsewhere you noted what the replacement value would be, which struck me as positive. Will we do that here? (That said, that is merely documentation, which while important, I would not prioritize since the important thing is to get the shape of the API right, which you seem to have done here.)

In reply to: 263108655 [](ancestors = 263108655)

Shahab is working on this, issue #2356

In reply to: 263109039 [](ancestors = 263109039,263108655)

TomFinley · 2019-03-06T19:55:52Z

src/Microsoft.ML.StandardLearners/Standard/Online/AveragedPerceptron.cs

-            [Argument(ArgumentType.Multiple, HelpText = "Loss Function", ShortName = "loss", SortOrder = 50)]
-            public ISupportClassificationLossFactory LossFunction = new HingeLoss.Options();
+            [Argument(ArgumentType.Multiple, Name = "LossFunction", HelpText = "Loss Function", ShortName = "loss", SortOrder = 50)]
+            internal ISupportClassificationLossFactory LossFunctionFactory1 = new HingeLoss.Options();


LossFunctionFactory1 [](start = 55, length = 20)

Was the 1 here intentional? #Resolved

I guess it was... would it be possible to give it a more clear name? As far as I can tell the choice of 1 was just an arbitrary value so that it didn't conflict with the existing LossFunctionFactory property, but ...

In reply to: 263109093 [](ancestors = 263109093)

Ivanidzo4ka · 2019-03-08T18:42:27Z

    public sealed class Options : ISupportSdcaClassificationLossFactory, ISupportClassificationLossFactory

why it's still public? #Closed

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:167 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka · 2019-03-08T18:42:41Z

    public sealed class Options : ISupportSdcaClassificationLossFactory, ISupportClassificationLossFactory

why it's still public? #Closed

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:239 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka · 2019-03-08T18:43:07Z

    public sealed class Options : ISupportClassificationLossFactory

why it's still public? #Closed

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:338 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka · 2019-03-08T18:43:38Z

    public ExpLoss(Options options)

why this public, and we don't have constructor to just pass beta? #Resolved

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:350 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka · 2019-03-08T18:44:07Z

    public sealed class Options : ISupportRegressionLossFactory

why it's still public? #Closed

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:445 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka · 2019-03-08T18:44:17Z

    public TweedieLoss(Options options)

why it's still public? #Closed

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:461 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka · 2019-03-09T00:52:41Z

    private sealed class TrivialFactory : ISupportRegressionLossFactory

Can we kill it? #Closed

Refers to: src/Microsoft.ML.StandardLearners/Standard/Online/OnlineGradientDescent.cs:123 in 2ba494d. [](commit_id = 2ba494d, deletion_comment = False)

Ivanidzo4ka · 2019-03-11T17:43:17Z

    public ExpLoss(Options options)

Please add constructor in which user can specify Beta.

In reply to: 471033031 [](ancestors = 471033031)

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:350 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

Ivanidzo4ka

ganik · 2019-03-11T17:53:55Z

    public ExpLoss(Options options)

added

In reply to: 471646221 [](ancestors = 471646221,471033031)

Refers to: src/Microsoft.ML.Data/Utils/LossFunctions.cs:350 in dd4b6f8. [](commit_id = dd4b6f8, deletion_comment = False)

TomFinley

Thank you @ganik !!

ganik and others added 9 commits February 26, 2019 14:17

Merge pull request #1 from dotnet/master

e3fe31e

Merge

Merge pull request #2 from dotnet/master

1fefaa0

Merge

Merge pull request #3 from dotnet/master

fcf85a2

Merge

Merge pull request #4 from dotnet/master

22dbc78

Merge

Hide ISupportSdcaClassificationLossFactory & ISupportSdcaRegressionLo…

ddd89fa

…ssFactory interfaces

fixes for more loss functions

1405be3

Merge pull request #6 from dotnet/master

5992842

Merge

Merge

bedd7f5

fix samples build

24c52e1

fix tests

21ea513

wschin reviewed Mar 5, 2019

View reviewed changes

fix test

400f552

wschin reviewed Mar 5, 2019

View reviewed changes

remove default Options() constructor

56b1af5

ganik requested review from TomFinley and Ivanidzo4ka March 6, 2019 00:15

TomFinley reviewed Mar 6, 2019

View reviewed changes

ganik added 3 commits March 6, 2019 17:14

Double -> float

5bd6e43

rename

8673943

to properties

dd4b6f8

fix comments

2ba494d

Ivanidzo4ka requested a review from wschin March 9, 2019 00:51

remove TrivialFactory

ffa170b

Ivanidzo4ka approved these changes Mar 11, 2019

View reviewed changes

add ExpLoss constructor

b4ec44f

TomFinley approved these changes Mar 11, 2019

View reviewed changes

This was referenced Mar 11, 2019

Fixing inconsistency in usage of LossFunction #2913

Closed

Fixing inconsistency in usage of LossFunction #2915

Merged

ganik closed this Mar 11, 2019

ganik deleted the ganik/ep2 branch April 1, 2019 20:55

ghost locked as resolved and limited conversation to collaborators Mar 23, 2022

Fixing inconsistency in usage of LossFunction #2856

Fixing inconsistency in usage of LossFunction #2856

Conversation

ganik commented Mar 5, 2019

Ivanidzo4ka commented Mar 5, 2019 • edited by ganik Loading

Ivanidzo4ka commented Mar 5, 2019 • edited by ganik Loading

Ivanidzo4ka commented Mar 5, 2019 • edited Loading

ganik commented Mar 5, 2019

ganik commented Mar 5, 2019

ganik commented Mar 5, 2019

wschin commented Mar 5, 2019 • edited by ganik Loading

wschin commented Mar 5, 2019 • edited by ganik Loading

wschin commented Mar 5, 2019 • edited by ganik Loading

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Mar 5, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin commented Mar 5, 2019 • edited by ganik Loading

ganik commented Mar 5, 2019

ganik commented Mar 5, 2019

codecov bot commented Mar 6, 2019 • edited Loading

Codecov Report

TomFinley commented Mar 6, 2019

TomFinley Mar 6, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomFinley Mar 6, 2019 • edited by ganik Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka commented Mar 8, 2019 • edited Loading

Ivanidzo4ka commented Mar 8, 2019 • edited Loading

Ivanidzo4ka commented Mar 8, 2019 • edited Loading

Ivanidzo4ka commented Mar 8, 2019 • edited by ganik Loading

Ivanidzo4ka commented Mar 8, 2019 • edited Loading

Ivanidzo4ka commented Mar 8, 2019 • edited Loading

Ivanidzo4ka commented Mar 9, 2019 • edited Loading

Ivanidzo4ka commented Mar 11, 2019

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

ganik commented Mar 11, 2019

TomFinley left a comment

Choose a reason for hiding this comment

Ivanidzo4ka commented Mar 5, 2019 •

edited by ganik

Loading

Ivanidzo4ka commented Mar 5, 2019 •

edited by ganik

Loading

Ivanidzo4ka commented Mar 5, 2019 •

edited

Loading

wschin commented Mar 5, 2019 •

edited by ganik

Loading

wschin commented Mar 5, 2019 •

edited by ganik

Loading

wschin commented Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin Mar 5, 2019 •

edited by ganik

Loading

wschin commented Mar 5, 2019 •

edited by ganik

Loading

codecov bot commented Mar 6, 2019 •

edited

Loading

TomFinley Mar 6, 2019 •

edited by ganik

Loading

TomFinley Mar 6, 2019 •

edited by ganik

Loading

Ivanidzo4ka commented Mar 8, 2019 •

edited

Loading

Ivanidzo4ka commented Mar 8, 2019 •

edited

Loading

Ivanidzo4ka commented Mar 8, 2019 •

edited

Loading

Ivanidzo4ka commented Mar 8, 2019 •

edited by ganik

Loading

Ivanidzo4ka commented Mar 8, 2019 •

edited

Loading

Ivanidzo4ka commented Mar 8, 2019 •

edited

Loading

Ivanidzo4ka commented Mar 9, 2019 •

edited

Loading