Generate new version for each DX Compiler build #2200

hekota · 2019-05-21T22:06:12Z

Added generating of new version for each DX Compiler build.

There are 3 kinds of version:

Official build
Built by using hctbuild -official. The version is based on the current DXIL version, latest official release and a number of commits since then. The format is dxil_major.dxil_minor.release_no.commit_count. For example a current official version would be something like 1.5.1905.42. The latest release information is read from utils\version\latest-release.json. The 1905 corresponds to dxil-2019-05-16 release branch and 42 is the number of commits since that release branch was created. For master branch the commit_count will be incremented by 10000 to distinguish it from stabilized official release branch builds. So the current official version of master would be someting like 1.5.1905.10042.
Dev build
Build by using hctbuild with no other version-related option. The format is dxil_major.dxil_minor.0.commit_count where commit_count is the number of total commits since the beginning of the project.
Fixed version build
Build by using hctbuild -fv. Enables overriding of the version information. The fixed version is read from utils\version\version.inc. Location of the version file can be overriden by -fvloc option on hctbuild.

In addition to the numbered version the product version string on the binaries will also include branch name and last commit sha - "1.5.1905.10042 (master, 47e31c8a)". This product version string is included in dxc -? output.

AppVeyorBot · 2019-05-22T00:07:19Z

❌ Build DirectXShaderCompiler 1.0.1898 failed (commit 24dc945 by @hekota)

AppVeyorBot · 2019-05-22T14:19:46Z

✅ Build DirectXShaderCompiler 1.0.1900 completed (commit 24dc945 by @hekota)

AppVeyorBot · 2019-05-24T23:12:04Z

✅ Build DirectXShaderCompiler 1.0.1919 completed (commit f1d5871 by @hekota)

AppVeyorBot · 2019-05-28T20:05:24Z

✅ Build DirectXShaderCompiler 1.0.1930 completed (commit 481e851 by @hekota)

AppVeyorBot · 2019-05-31T21:09:47Z

✅ Build DirectXShaderCompiler 1.0.1953 completed (commit de1d14b by @hekota)

AppVeyorBot · 2019-06-25T20:46:09Z

✅ Build DirectXShaderCompiler 1.0.2069 completed (commit c2f8c61 by @hekota)

MrTrillian · 2019-06-25T21:33:58Z

What happened for there to be so many other commits in here? Did something go wrong? I don't think you want to merge as is, and it's difficult to review your specific changes.

hekota · 2019-06-25T22:04:52Z

I need to rebase it.


        Generate new version for each DX Compiler build

The version format is: 14.YYYY.1mmdd.1HHMM "14" stands for DXIL 1.4. Product version string includes name of a branch and last commit sha. Enable use of fixed version (read from a file). New -fvloc flag on hctbuild.cmd for specifying custom version file


        Report product version with branch name and hash information in "dxc -?"


        Minor fixes and improvements


        Comment fix


        Read DXIL major and minor version from include/dxc/dxil/dxilconstants.h

Change version to dxil_major.dxil_minor.xxx.xxx, where xxx is currently derived from unix epoch time but that might change). In "dxc.exe /?" detect old dev build version 3.7.0.0 and output the commit info and commit count (in case dxc.exe works on top of older dxcompiler.dll) Code review feedback


        Determine location of DxilConstants.h from gen_version.py location


        Generate version based on the latest release and commit count.

New -official flag on hctbuild will embed an official version to the dll. This version will be based on the latest release number (read from latest-release.json) and the number of commits since we branched for this release. For master branch we'll add 10000 to the commit count. Dev builds will have a version 1.5.0.commitcount (1.5. is a DXIL version and comes from latest-release.json)

AppVeyorBot · 2019-06-25T22:55:14Z

✅ Build DirectXShaderCompiler 1.0.2071 completed (commit 90565a9 by @hekota)


        Get Python from PYTHON_EXECUTABLE variable (fixes Travis build)

tools/clang/tools/dxc/dxclib/dxc.cpp

MrTrillian · 2019-06-27T18:59:46Z

utils/version/version.inc

@@ -0,0 +1,52 @@
+#pragma once


Should we be checking in this file? It will be generated by the build, won't it?

There is a -fv option on hctbuild that enables overiding of the generated version and this is the default file it reads the version info from. The location can be overriden by the new -fvloc option but -fv needs to work on its own (I believe AppVeyor uses it and that's how we versioned the releases until now). It's a new file because the fixed version was previously specified as strings in AddLLVM.cmake. Now we don't need to touch cmake files to change the version.

Ok! So the file generated by the build has the same format but will be somewhere else?

Same format and same location (<build_dir>\utils\version), but it is either going to be generated or copied from here.

MrTrillian · 2019-06-27T19:02:32Z

utils/version/gen_version.py

@@ -0,0 +1,107 @@
+import argparse


I'd like a header comment summarizing the version scheme implemented here. We have no documentation for it outside of the code itself, and while not too complex, it still needs reading through. This could be a place to write why we chose that scheme too, to leave a paper trail.

Yes, I've just finished updating the PR description and I will copy that here.

MrTrillian

Looking good! Can you just check the few comments I've left?

AppVeyorBot · 2019-06-27T19:43:48Z

✅ Build DirectXShaderCompiler 1.0.2082 completed (commit 4aff2fa by @hekota)


        Code review feedback

gen_version.py - Change git.exe to git for Linux runs

MrTrillian · 2019-06-27T21:36:21Z

tools/clang/tools/dxc/dxclib/dxc.cpp

+  DWORD dwVerHnd = 0;
+  DWORD size = GetFileVersionInfoSize(dllPath, &dwVerHnd);
+  if (size == 0) return false;
+  std::unique_ptr<int[]> VfInfo(new int[size/sizeof(int)]);


This fix is not correct as it will round down the number of bytes to the nearest multiple of four, but then you still pass "size" as the size. You can make this a char or BYTE, all of the functions you pass this as an argument to take a void* anyways so they don't care. What was the reason for making it an int?

Right, sorry ;) There is no reason for this to be int. I have copied that from another place in this file. I'll fix both.

AppVeyorBot · 2019-06-27T22:08:02Z

✅ Build DirectXShaderCompiler 1.0.2084 completed (commit 3249272 by @hekota)


        Fix array type and size

AppVeyorBot · 2019-06-27T23:07:35Z

✅ Build DirectXShaderCompiler 1.0.2085 completed (commit ac0c6eb by @hekota)

sunillnaik · 2019-07-23T16:54:42Z

Since I dont have git setup, I only have source, how should I skip this gen_version.py execution in the build?

Building Custom Rule D:/MSDXCCompiler/dev/src/utils/version/CMakeLists.txt
 CMake does not need to re-run because D:/MSDXCCompiler/dev/build/win_x64/Release/utils/version/CMakeFiles/generate.stamp is up-to-date.
 "Generating version"
 fatal: not a git repository (or any of the parent directories): .git
 Traceback (most recent call last):
   File "D:/MSDXCCompiler/dev/src/utils/version/gen_version.py", line 132, in <module>
     main()

hekota · 2019-07-23T19:20:41Z

Ok, try building with the -fv flag:

hctbuild -fv

That will skip the version generating and use a fixed version file.

hekota requested review from MrTrillian and tex3d May 21, 2019

hekota requested a review from jpporto May 23, 2019

hekota changed the title ~~Generate new version for each DX Compiler build~~ (do not merge) Generate new version for each DX Compiler build May 24, 2019

hekota force-pushed the hekota:autogen-version branch 3 times, most recently from 8c70330 to 3c992f8 May 24, 2019

hekota force-pushed the hekota:autogen-version branch from 3c992f8 to c55568a May 28, 2019

hekota added 7 commits May 20, 2019

Report product version with branch name and hash information in "dxc -?"

0d648ed

Minor fixes and improvements

0e7154e

Comment fix

f827087

Determine location of DxilConstants.h from gen_version.py location

909292d

hekota force-pushed the hekota:autogen-version branch from 6f4c2bf to ab205b4 Jun 25, 2019

hekota changed the title ~~(do not merge) Generate new version for each DX Compiler build~~ Generate new version for each DX Compiler build Jun 25, 2019

Get Python from PYTHON_EXECUTABLE variable (fixes Travis build)

Loading status checks…

50ffa8b

MrTrillian reviewed Jun 27, 2019

View changes

tools/clang/tools/dxc/dxclib/dxc.cpp Outdated Show resolved Hide resolved

MrTrillian reviewed Jun 27, 2019

View changes

tools/clang/tools/dxc/dxclib/dxc.cpp Show resolved Hide resolved

MrTrillian reviewed Jun 27, 2019

View changes

MrTrillian approved these changes Jun 27, 2019

View changes

Code review feedback

Loading status checks…

cdab47b

gen_version.py - Change git.exe to git for Linux runs

MrTrillian reviewed Jun 27, 2019

View changes

Fix array type and size

Loading status checks…

7ce8636

hekota merged commit 5edbabb into microsoft:master Jun 28, 2019
3 checks passed

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

license/cla All CLA requirements met.
Details

hekota deleted the hekota:autogen-version branch Jul 12, 2019

Aug	SEP	Oct
	10
2019	2020	2021

microsoft / DirectXShaderCompiler

Generate new version for each DX Compiler build #2200

Generate new version for each DX Compiler build #2200

hekota commented May 21, 2019 •

edited

AppVeyorBot commented May 22, 2019

AppVeyorBot commented May 22, 2019

AppVeyorBot commented May 24, 2019

AppVeyorBot commented May 28, 2019

AppVeyorBot commented May 31, 2019

AppVeyorBot commented Jun 25, 2019

MrTrillian commented Jun 25, 2019

hekota commented Jun 25, 2019

AppVeyorBot commented Jun 25, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

MrTrillian left a comment

AppVeyorBot commented Jun 27, 2019

This comment has been minimized.

This comment has been minimized.

AppVeyorBot commented Jun 27, 2019

AppVeyorBot commented Jun 27, 2019

This comment has been minimized.

sunillnaik commented on `cdab47b` Jul 23, 2019

This comment has been minimized.

hekota replied Jul 23, 2019

microsoft / DirectXShaderCompiler

Join GitHub today

Generate new version for each DX Compiler build #2200

Generate new version for each DX Compiler build #2200

Conversation

hekota commented May 21, 2019 • edited

AppVeyorBot commented May 22, 2019

AppVeyorBot commented May 22, 2019

AppVeyorBot commented May 24, 2019

AppVeyorBot commented May 28, 2019

AppVeyorBot commented May 31, 2019

AppVeyorBot commented Jun 25, 2019

MrTrillian commented Jun 25, 2019

hekota commented Jun 25, 2019

AppVeyorBot commented Jun 25, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

MrTrillian left a comment

AppVeyorBot commented Jun 27, 2019

This comment has been minimized.

This comment has been minimized.

AppVeyorBot commented Jun 27, 2019

AppVeyorBot commented Jun 27, 2019

This comment has been minimized.

sunillnaik commented on cdab47b Jul 23, 2019

This comment has been minimized.

hekota replied Jul 23, 2019

hekota commented May 21, 2019 •

edited

sunillnaik commented on `cdab47b` Jul 23, 2019