Skip to content

Commit aefdfae

Browse files
YangqingRob Kunkle
Yangqing
authored and
Rob Kunkle
committed
gflags improvement to allow CAFFE2_EXPORTS (pytorch#10444)
Summary: Explanation copied from code: // Motivation about the gflags wrapper: // (1) We would need to make sure that the gflags version and the non-gflags // version of Caffe2 are going to expose the same flags abstraction. One should // explicitly use caffe2::FLAGS_flag_name to access the flags. // (2) For flag names, it is recommended to start with caffe2_ to distinguish it // from regular gflags flags. For example, do // CAFFE2_DEFINE_BOOL(caffe2_my_flag, true, "An example"); // to allow one to use caffe2::FLAGS_caffe2_my_flag. // (3) Gflags has a design issue that does not properly expose the global flags, // if one builds the library with -fvisibility=hidden. The current gflags (as of // Aug 2018) only deals with the Windows case using dllexport, and not the Linux // counterparts. As a result, we will explciitly use CAFFE2_EXPORT to export the // flags defined in Caffe2. This is done via a global reference, so the flag // itself is not duplicated - under the hood it is the same global gflags flag. Pull Request resolved: pytorch#10444 Differential Revision: D9296726 Pulled By: Yangqing fbshipit-source-id: a867d67260255cc46bf0a928122ff71a575d3966
1 parent b12d44e commit aefdfae

File tree

3 files changed

+120
-49
lines changed

3 files changed

+120
-49
lines changed

‎caffe2/core/flags.h

Lines changed: 80 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,12 @@ namespace caffe2 {
2626
/**
2727
* Sets the usage message when a commandline tool is called with "--help".
2828
*/
29-
void SetUsageMessage(const string& str);
29+
CAFFE2_API void SetUsageMessage(const string& str);
3030

3131
/**
3232
* Returns the usage message for the commandline tool set by SetUsageMessage.
3333
*/
34-
const char* UsageMessage();
34+
CAFFE2_API const char* UsageMessage();
3535

3636
/**
3737
* Parses the commandline flags.
@@ -41,11 +41,11 @@ const char* UsageMessage();
4141
* commandline args that caffe2 does not deal with. Note that following
4242
* convention, argv[0] contains the binary name and is not parsed.
4343
*/
44-
bool ParseCaffeCommandLineFlags(int* pargc, char*** pargv);
44+
CAFFE2_API bool ParseCaffeCommandLineFlags(int* pargc, char*** pargv);
4545
/**
4646
* Checks if the commandline flags has already been passed.
4747
*/
48-
bool CommandLineFlagsHasBeenParsed();
48+
CAFFE2_API bool CommandLineFlagsHasBeenParsed();
4949

5050
} // namespace caffe2
5151

@@ -56,6 +56,10 @@ bool CommandLineFlagsHasBeenParsed();
5656

5757
#ifdef CAFFE2_USE_GFLAGS
5858

59+
////////////////////////////////////////////////////////////////////////////////
60+
// Begin gflags section: most functions are basically rerouted to gflags.
61+
////////////////////////////////////////////////////////////////////////////////
62+
5963
#include <gflags/gflags.h>
6064

6165
// gflags before 2.0 uses namespace google and after 2.1 uses namespace gflags.
@@ -64,41 +68,70 @@ bool CommandLineFlagsHasBeenParsed();
6468
namespace gflags = google;
6569
#endif // GFLAGS_GFLAGS_H_
6670

67-
#define CAFFE2_GFLAGS_DEF_WRAPPER(type, name, default_value, help_str) \
71+
// Motivation about the gflags wrapper:
72+
// (1) We would need to make sure that the gflags version and the non-gflags
73+
// version of Caffe2 are going to expose the same flags abstraction. One should
74+
// explicitly use caffe2::FLAGS_flag_name to access the flags.
75+
// (2) For flag names, it is recommended to start with caffe2_ to distinguish it
76+
// from regular gflags flags. For example, do
77+
// CAFFE2_DEFINE_BOOL(caffe2_my_flag, true, "An example");
78+
// to allow one to use caffe2::FLAGS_caffe2_my_flag.
79+
// (3) Gflags has a design issue that does not properly expose the global flags,
80+
// if one builds the library with -fvisibility=hidden. The current gflags (as of
81+
// Aug 2018) only deals with the Windows case using dllexport, and not the Linux
82+
// counterparts. As a result, we will explciitly use CAFFE2_EXPORT to export the
83+
// flags defined in Caffe2. This is done via a global reference, so the flag
84+
// itself is not duplicated - under the hood it is the same global gflags flag.
85+
#define CAFFE2_GFLAGS_DEF_WRAPPER( \
86+
type, real_type, name, default_value, help_str) \
6887
DEFINE_##type(name, default_value, help_str); \
6988
namespace caffe2 { \
70-
using ::FLAGS_##name; \
89+
CAFFE2_EXPORT real_type& FLAGS_##name = ::FLAGS_##name; \
7190
}
7291

7392
#define CAFFE2_DEFINE_int(name, default_value, help_str) \
74-
CAFFE2_GFLAGS_DEF_WRAPPER(int32, name, default_value, help_str)
93+
CAFFE2_GFLAGS_DEF_WRAPPER(int32, gflags::int32, name, default_value, help_str)
7594
#define CAFFE2_DEFINE_int64(name, default_value, help_str) \
76-
CAFFE2_GFLAGS_DEF_WRAPPER(int64, name, default_value, help_str)
95+
CAFFE2_GFLAGS_DEF_WRAPPER(int64, gflags::int64, name, default_value, help_str)
7796
#define CAFFE2_DEFINE_double(name, default_value, help_str) \
78-
CAFFE2_GFLAGS_DEF_WRAPPER(double, name, default_value, help_str)
97+
CAFFE2_GFLAGS_DEF_WRAPPER(double, double, name, default_value, help_str)
7998
#define CAFFE2_DEFINE_bool(name, default_value, help_str) \
80-
CAFFE2_GFLAGS_DEF_WRAPPER(bool, name, default_value, help_str)
81-
#define CAFFE2_DEFINE_string(name, default_value, help_str) \
82-
CAFFE2_GFLAGS_DEF_WRAPPER(string, name, default_value, help_str)
99+
CAFFE2_GFLAGS_DEF_WRAPPER(bool, bool, name, default_value, help_str)
100+
#define CAFFE2_DEFINE_string(name, default_value, help_str) \
101+
CAFFE2_GFLAGS_DEF_WRAPPER( \
102+
string, ::fLS::clstring, name, default_value, help_str)
83103

84104
// DECLARE_typed_var should be used in header files and in the global namespace.
85-
#define CAFFE2_GFLAGS_DECLARE_WRAPPER(type, name) \
86-
DECLARE_##type(name); \
87-
namespace caffe2 { \
88-
using ::FLAGS_##name; \
105+
#define CAFFE2_GFLAGS_DECLARE_WRAPPER(type, real_type, name) \
106+
DECLARE_##type(name); \
107+
namespace caffe2 { \
108+
extern real_type& FLAGS_##name ; \
89109
} // namespace caffe2
90110

91-
#define CAFFE2_DECLARE_int(name) CAFFE2_GFLAGS_DECLARE_WRAPPER(int32, name)
92-
#define CAFFE2_DECLARE_int64(name) CAFFE2_GFLAGS_DECLARE_WRAPPER(int64, name)
93-
#define CAFFE2_DECLARE_double(name) CAFFE2_GFLAGS_DECLARE_WRAPPER(double, name)
94-
#define CAFFE2_DECLARE_bool(name) CAFFE2_GFLAGS_DECLARE_WRAPPER(bool, name)
95-
#define CAFFE2_DECLARE_string(name) CAFFE2_GFLAGS_DECLARE_WRAPPER(string, name)
111+
#define CAFFE2_DECLARE_int(name) \
112+
CAFFE2_GFLAGS_DECLARE_WRAPPER(int32, gflags::int32, name)
113+
#define CAFFE2_DECLARE_int64(name) \
114+
CAFFE2_GFLAGS_DECLARE_WRAPPER(int64, gflags::int64, name)
115+
#define CAFFE2_DECLARE_double(name) \
116+
CAFFE2_GFLAGS_DECLARE_WRAPPER(double, double, name)
117+
#define CAFFE2_DECLARE_bool(name) \
118+
CAFFE2_GFLAGS_DECLARE_WRAPPER(bool, bool, name)
119+
#define CAFFE2_DECLARE_string(name) \
120+
CAFFE2_GFLAGS_DECLARE_WRAPPER(string, ::fLS::clstring, name)
121+
122+
////////////////////////////////////////////////////////////////////////////////
123+
// End gflags section.
124+
////////////////////////////////////////////////////////////////////////////////
96125

97126
#else // CAFFE2_USE_GFLAGS
98127

128+
////////////////////////////////////////////////////////////////////////////////
129+
// Begin non-gflags section: providing equivalent functionality.
130+
////////////////////////////////////////////////////////////////////////////////
131+
99132
namespace caffe2 {
100133

101-
class Caffe2FlagParser {
134+
class CAFFE2_API Caffe2FlagParser {
102135
public:
103136
Caffe2FlagParser() {}
104137
bool success() { return success_; }
@@ -117,39 +150,39 @@ CAFFE_DECLARE_REGISTRY(Caffe2FlagsRegistry, Caffe2FlagParser, const string&);
117150
// write the CAFFE2_DEFINE_* and CAFFE2_DECLARE_* macros outside any namespace
118151
// as well.
119152

120-
#define CAFFE2_DEFINE_typed_var(type, name, default_value, help_str) \
121-
namespace caffe2 { \
122-
CAFFE2_EXPORT type FLAGS_##name = default_value; \
123-
namespace { \
124-
class Caffe2FlagParser_##name : public Caffe2FlagParser { \
125-
public: \
126-
explicit Caffe2FlagParser_##name(const string& content) { \
127-
success_ = Caffe2FlagParser::Parse<type>(content, &FLAGS_##name); \
128-
} \
129-
}; \
130-
} \
131-
RegistererCaffe2FlagsRegistry g_Caffe2FlagsRegistry_##name( \
132-
#name, \
133-
Caffe2FlagsRegistry(), \
134-
RegistererCaffe2FlagsRegistry::DefaultCreator<Caffe2FlagParser_##name>, \
135-
"(" #type ", default " #default_value ") " help_str); \
153+
#define CAFFE2_DEFINE_typed_var(type, name, default_value, help_str) \
154+
namespace caffe2 { \
155+
CAFFE2_EXPORT type FLAGS_##name = default_value; \
156+
namespace { \
157+
class Caffe2FlagParser_##name : public Caffe2FlagParser { \
158+
public: \
159+
explicit Caffe2FlagParser_##name(const string& content) { \
160+
success_ = Caffe2FlagParser::Parse<type>(content, &FLAGS_##name); \
161+
} \
162+
}; \
163+
} \
164+
RegistererCaffe2FlagsRegistry g_Caffe2FlagsRegistry_##name( \
165+
#name, \
166+
Caffe2FlagsRegistry(), \
167+
RegistererCaffe2FlagsRegistry::DefaultCreator<Caffe2FlagParser_##name>, \
168+
"(" #type ", default " #default_value ") " help_str); \
136169
}
137170

138171
#define CAFFE2_DEFINE_int(name, default_value, help_str) \
139172
CAFFE2_DEFINE_typed_var(int, name, default_value, help_str)
140-
#define CAFFE2_DEFINE_int64(name, default_value, help_str) \
173+
#define CAFFE2_DEFINE_int64(name, default_value, help_str) \
141174
CAFFE2_DEFINE_typed_var(int64_t, name, default_value, help_str)
142-
#define CAFFE2_DEFINE_double(name, default_value, help_str) \
175+
#define CAFFE2_DEFINE_double(name, default_value, help_str) \
143176
CAFFE2_DEFINE_typed_var(double, name, default_value, help_str)
144177
#define CAFFE2_DEFINE_bool(name, default_value, help_str) \
145178
CAFFE2_DEFINE_typed_var(bool, name, default_value, help_str)
146179
#define CAFFE2_DEFINE_string(name, default_value, help_str) \
147180
CAFFE2_DEFINE_typed_var(string, name, default_value, help_str)
148181

149182
// DECLARE_typed_var should be used in header files and in the global namespace.
150-
#define CAFFE2_DECLARE_typed_var(type, name) \
151-
namespace caffe2 { \
152-
CAFFE2_IMPORT extern type FLAGS_##name; \
183+
#define CAFFE2_DECLARE_typed_var(type, name) \
184+
namespace caffe2 { \
185+
CAFFE2_IMPORT extern type FLAGS_##name; \
153186
} // namespace caffe2
154187

155188
#define CAFFE2_DECLARE_int(name) CAFFE2_DECLARE_typed_var(int, name)
@@ -158,6 +191,10 @@ CAFFE_DECLARE_REGISTRY(Caffe2FlagsRegistry, Caffe2FlagParser, const string&);
158191
#define CAFFE2_DECLARE_bool(name) CAFFE2_DECLARE_typed_var(bool, name)
159192
#define CAFFE2_DECLARE_string(name) CAFFE2_DECLARE_typed_var(string, name)
160193

194+
////////////////////////////////////////////////////////////////////////////////
195+
// End non-gflags section.
196+
////////////////////////////////////////////////////////////////////////////////
197+
161198
#endif // CAFFE2_USE_GFLAGS
162199

163200
#endif // CAFFE2_CORE_FLAGS_H_

‎caffe2/core/flags_test.cc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#include <gtest/gtest.h>
2+
#include "caffe2/core/macros.h"
3+
#include "caffe2/core/flags.h"
4+
#include "caffe2/core/logging.h"
5+
6+
CAFFE2_DEFINE_bool(caffe2_flags_test_only_flag, true, "Only used in test.");
7+
8+
namespace caffe2 {
9+
10+
TEST(FlagsTest, TestGflagsCorrectness) {
11+
#ifdef CAFFE2_USE_GFLAGS
12+
EXPECT_EQ(FLAGS_caffe2_flags_test_only_flag, true);
13+
EXPECT_EQ(::FLAGS_caffe2_flags_test_only_flag, true);
14+
// Change the caffe2 namespace and check global
15+
FLAGS_caffe2_flags_test_only_flag = false;
16+
EXPECT_EQ(FLAGS_caffe2_flags_test_only_flag, false);
17+
EXPECT_EQ(::FLAGS_caffe2_flags_test_only_flag, false);
18+
// Change global and check caffe2 namespace
19+
::FLAGS_caffe2_flags_test_only_flag = true;
20+
EXPECT_EQ(FLAGS_caffe2_flags_test_only_flag, true);
21+
EXPECT_EQ(::FLAGS_caffe2_flags_test_only_flag, true);
22+
#else // CAFFE2_USE_GFLAGS
23+
LOG(INFO) << "Caffe2 is not built with gflags. Nothing to test here.";
24+
#endif
25+
}
26+
27+
} // namespace caffe2

‎caffe2/core/logging.cc

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -69,20 +69,27 @@ std::function<void(const OperatorDef&)> GetOperatorLogger() {
6969
#ifdef CAFFE2_USE_GOOGLE_GLOG
7070

7171
#ifdef CAFFE2_USE_GFLAGS
72+
// When GLOG depends on GFLAGS, these variables are being defined in GLOG
73+
// directly via the GFLAGS definition, so we will use DECLARE_* to declare
74+
// them, and use them in Caffe2.
7275
// GLOG's minloglevel
73-
CAFFE2_DECLARE_int(minloglevel);
76+
DECLARE_int32(minloglevel);
7477
// GLOG's verbose log value.
75-
CAFFE2_DECLARE_int(v);
78+
DECLARE_int32(v);
7679
// GLOG's logtostderr value
77-
CAFFE2_DECLARE_bool(logtostderr);
78-
79-
#else
80+
DECLARE_bool(logtostderr);
81+
#endif // CAFFE2_USE_GFLAGS
8082

83+
// Provide easy access to the above variables, regardless whether GLOG is
84+
// dependent on GFLAGS or not. Note that the namespace (fLI, fLB) is actually
85+
// consistent between GLOG and GFLAGS, so we can do the below declaration
86+
// consistently.
87+
namespace caffe2 {
8188
using fLI::FLAGS_minloglevel;
8289
using fLI::FLAGS_v;
8390
using fLB::FLAGS_logtostderr;
91+
} // namespace caffe2
8492

85-
#endif // CAFFE2_USE_GFLAGS
8693

8794
CAFFE2_DEFINE_int(caffe2_log_level, google::GLOG_ERROR,
8895
"The minimum log level that caffe2 will output.");

0 commit comments

Comments
 (0)