linux/scripts/coccicheck

258 lines
7.2 KiB

coccicheck: enable parmap support Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported and use it only if your number of processors detected is > 1. If parmap is enabled also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. stderr is currently sent to /dev/null, addressing a way to capture that will be addressed next. If --jobs is not supported we fallback to the old mechanism. We expect to deprecate the old mechanism as soon as we can get confirmation all users are ready. While at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Extend Documentation/coccinelle.txt as well. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v4: o expand Documentation/coccinelle.txt to reflect parmap support info o update commit log to reflect what we actually do now with stderr o split out DEBUG_FILE use into another patch o detect number of CPUs and if its 1 then skip parmap support, note that if you still support parmap, but have 1 CPU you will also go through the new branches, so the old complex multithreaded process is skipped as well. v3: o move USE_JOBS to avoid being overriden v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-29 15:14:53 -07:00
coccinelle: fix parallel build with CHECK=scripts/coccicheck [ Upstream commit d7059ca0147adcd495f3c5b41f260e1ac55bb679 ] The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of "coccicheck failed" error messages. Julia Lawall explained the Coccinelle behavior as follows: "The problem on the Coccinelle side is that it uses a subdirectory with the name of the semantic patch to store standard output and standard error for the different threads. I didn't want to use a name with the pid, so that one could easily find this information while Coccinelle is running. Normally the subdirectory is cleaned up when Coccinelle completes, so there is only one of them at a time. Maybe it is best to just add the pid. There is the risk that these subdirectories will accumulate if Coccinelle crashes in a way such that they don't get cleaned up, but Coccinelle could print a warning if it detects this case, rather than failing." When scripts/coccicheck is used as CHECK tool and -j option is given to Make, the whole of build process runs in parallel. So, multiple processes try to get access to the same subdirectory. I notice spatch creates the subdirectory only when it runs in parallel (i.e. --jobs <N> is given and <N> is greater than 1). Setting NPROC=1 is a reasonable solution; spatch does not create the subdirectory. Besides, ONLINE=1 mode takes a single file input for each spatch invocation, so there is no reason to parallelize it in the first place. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Acked-by: Julia Lawall <Julia.Lawall@lip6.fr> Signed-off-by: Sasha Levin <alexander.levin@verizon.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-14 20:38:07 +09:00
coccinelle: fix parallel build with CHECK=scripts/coccicheck [ Upstream commit d7059ca0147adcd495f3c5b41f260e1ac55bb679 ] The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of "coccicheck failed" error messages. Julia Lawall explained the Coccinelle behavior as follows: "The problem on the Coccinelle side is that it uses a subdirectory with the name of the semantic patch to store standard output and standard error for the different threads. I didn't want to use a name with the pid, so that one could easily find this information while Coccinelle is running. Normally the subdirectory is cleaned up when Coccinelle completes, so there is only one of them at a time. Maybe it is best to just add the pid. There is the risk that these subdirectories will accumulate if Coccinelle crashes in a way such that they don't get cleaned up, but Coccinelle could print a warning if it detects this case, rather than failing." When scripts/coccicheck is used as CHECK tool and -j option is given to Make, the whole of build process runs in parallel. So, multiple processes try to get access to the same subdirectory. I notice spatch creates the subdirectory only when it runs in parallel (i.e. --jobs <N> is given and <N> is greater than 1). Setting NPROC=1 is a reasonable solution; spatch does not create the subdirectory. Besides, ONLINE=1 mode takes a single file input for each spatch invocation, so there is no reason to parallelize it in the first place. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Acked-by: Julia Lawall <Julia.Lawall@lip6.fr> Signed-off-by: Sasha Levin <alexander.levin@verizon.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-14 20:38:07 +09:00
coccicheck: enable parmap support Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported and use it only if your number of processors detected is > 1. If parmap is enabled also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. stderr is currently sent to /dev/null, addressing a way to capture that will be addressed next. If --jobs is not supported we fallback to the old mechanism. We expect to deprecate the old mechanism as soon as we can get confirmation all users are ready. While at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Extend Documentation/coccinelle.txt as well. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v4: o expand Documentation/coccinelle.txt to reflect parmap support info o update commit log to reflect what we actually do now with stderr o split out DEBUG_FILE use into another patch o detect number of CPUs and if its 1 then skip parmap support, note that if you still support parmap, but have 1 CPU you will also go through the new branches, so the old complex multithreaded process is skipped as well. v3: o move USE_JOBS to avoid being overriden v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-29 15:14:53 -07:00
coccicheck: enable parmap support Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported and use it only if your number of processors detected is > 1. If parmap is enabled also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. stderr is currently sent to /dev/null, addressing a way to capture that will be addressed next. If --jobs is not supported we fallback to the old mechanism. We expect to deprecate the old mechanism as soon as we can get confirmation all users are ready. While at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Extend Documentation/coccinelle.txt as well. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v4: o expand Documentation/coccinelle.txt to reflect parmap support info o update commit log to reflect what we actually do now with stderr o split out DEBUG_FILE use into another patch o detect number of CPUs and if its 1 then skip parmap support, note that if you still support parmap, but have 1 CPU you will also go through the new branches, so the old complex multithreaded process is skipped as well. v3: o move USE_JOBS to avoid being overriden v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-29 15:14:53 -07:00
coccicheck: enable parmap support Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported and use it only if your number of processors detected is > 1. If parmap is enabled also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. stderr is currently sent to /dev/null, addressing a way to capture that will be addressed next. If --jobs is not supported we fallback to the old mechanism. We expect to deprecate the old mechanism as soon as we can get confirmation all users are ready. While at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Extend Documentation/coccinelle.txt as well. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v4: o expand Documentation/coccinelle.txt to reflect parmap support info o update commit log to reflect what we actually do now with stderr o split out DEBUG_FILE use into another patch o detect number of CPUs and if its 1 then skip parmap support, note that if you still support parmap, but have 1 CPU you will also go through the new branches, so the old complex multithreaded process is skipped as well. v3: o move USE_JOBS to avoid being overriden v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-29 15:14:53 -07:00
coccicheck: enable parmap support Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported and use it only if your number of processors detected is > 1. If parmap is enabled also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. stderr is currently sent to /dev/null, addressing a way to capture that will be addressed next. If --jobs is not supported we fallback to the old mechanism. We expect to deprecate the old mechanism as soon as we can get confirmation all users are ready. While at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Extend Documentation/coccinelle.txt as well. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v4: o expand Documentation/coccinelle.txt to reflect parmap support info o update commit log to reflect what we actually do now with stderr o split out DEBUG_FILE use into another patch o detect number of CPUs and if its 1 then skip parmap support, note that if you still support parmap, but have 1 CPU you will also go through the new branches, so the old complex multithreaded process is skipped as well. v3: o move USE_JOBS to avoid being overriden v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> Acked-by: Nicolas Palix <nicolas.palix@imag.fr> Signed-off-by: Michal Marek <mmarek@suse.com>
2016-06-29 15:14:53 -07:00