Finalize story for libtest

One of the last remaining crates to have a stabilization story in the standard
distribution is the libtest library. This library currently provides the backing
infrastructure and test harness used when rustc generates a test execuable via
the `--test` command line flag.

It's well known that libtest is not the end-all-be-all of testing frameworks. It
is intentionally minimal and is currently quite conservative in its scope of
what it tries to accomplish as well as what it implements. A testament to this
is the fact that one very rarely writes `extern crate test`, and it almost means
that the stabilization story need not be considered for the crate at all! The
benchmarking feature of the compiler, however, is quite useful and is one of the
sole reasons for using `extern crate test`.

When benchmarking, there are a few primary interfaces to the libtest library
that are used:

* `test::Bencher` - the type for a benchmarking harness.
* `test::Bencher::iter` - a member function used to run a benchmark.
* `test::black_box` - a useful function to hinder optimizations and prevent a
  value from being optimized away.

These three pieces of information are the primary targets for the stabilization
in this commit. The rest of the testing infrastructure, while still quite useful
to some projects, is not in scope for stabilization at 1.0 and will remain
`#[experimental]` for now.

The benchmarking pieces have been moved to a new `rustc_bench` crate which will
be part of the standard distribution. In order to write a benchmark one will
need to import the crate via `extern crate rustc_bench` and otherwise all usage
remains the same. The purpose of this crate is to provide a clear area for these
benchmarking utilities as well as provide a clear name that it is *only*
intended for use via the compiler `#[bench]` attribute. The current interface is
quite minimal with only what's necessary as `#[stable]`.

It is most certainly a desire for the compiler to support other testing
frameworks other than the standard libtest. This form of infrastructure (be it a
plugin or a separate interface) is out of scope for 1.0, however, and this
commit does not attempt to resolve it.

Due to the removal of benchmarking items from libtest, this is a breaking
change. To update, rewrite imports of `extern crate test` to `extern crate
rustc_bench` and then rewrite all `use` statements as well. The public interface
of the `Bencher` struct has not changed.

[breaking-change]

Author: alexcrichton

mk/crates.mk

@@ -52,7 +52,7 @@
 TARGET_CRATES := libc std flate arena term \
                  serialize getopts collections test rand \
                  log regex graphviz core rbml alloc \
-                 unicode
+                 unicode rustc_bench
 RUSTC_CRATES := rustc rustc_typeck rustc_borrowck rustc_resolve rustc_driver \
                 rustc_trans rustc_back rustc_llvm
 HOST_CRATES := syntax $(RUSTC_CRATES) rustdoc fmt_macros
@@ -90,11 +90,13 @@ DEPS_term := std log
 DEPS_getopts := std
 DEPS_collections := core alloc unicode
 DEPS_num := std
-DEPS_test := std getopts serialize rbml term regex native:rust_test_helpers
+DEPS_test := std getopts serialize rbml term regex native:rust_test_helpers \
+		rustc_bench
 DEPS_rand := core
 DEPS_log := std regex
 DEPS_regex := std
 DEPS_fmt_macros = std
+DEPS_rustc_bench := std
 
 TOOL_DEPS_compiletest := test getopts
 TOOL_DEPS_rustdoc := rustdoc

src/doc/guide-testing.md

@@ -1,7 +1,7 @@
 % The Rust Testing Guide
 
 > Program testing can be a very effective way to show the presence of bugs, but
-> it is hopelessly inadequate for showing their absence. 
+> it is hopelessly inadequate for showing their absence.
 >
 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
 
@@ -310,7 +310,7 @@ extern crate adder;
 #[test]
 fn it_works() {
     assert_eq(4, adder::add_two(2));
-}   
+}
 ```
 
 This looks similar to our previous tests, but slightly different. We now have
@@ -442,7 +442,7 @@ code. Let's make our `src/lib.rs` look like this (comments elided):
 ```{rust,ignore}
 #![feature(globs)]
 
-extern crate test;
+extern crate rustc_bench;
 
 pub fn add_two(a: i32) -> i32 {
     a + 2
@@ -451,7 +451,7 @@ pub fn add_two(a: i32) -> i32 {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use test::Bencher;
+    use rustc_bench::Bencher;
 
     #[test]
     fn it_works() {
@@ -512,8 +512,8 @@ compiler might recognize that some calculation has no external effects and
 remove it entirely.
 
 ```{rust,ignore}
-extern crate test;
-use test::Bencher;
+extern crate rustc_bench;
+use rustc_bench::Bencher;
 
 #[bench]
 fn bench_xor_1000_ints(b: &mut Bencher) {
@@ -547,20 +547,19 @@ b.iter(|| {
 });
 ```
 
-Or, the other option is to call the generic `test::black_box` function, which
-is an opaque "black box" to the optimizer and so forces it to consider any
-argument as used.
+Or, the other option is to call the generic `rustc_bench::black_box` function,
+which is an opaque "black box" to the optimizer and so forces it to consider
+any argument as used.
 
 ```rust
-extern crate test;
-
+extern crate rustc_bench;
 # fn main() {
 # struct X;
 # impl X { fn iter<T, F>(&self, _: F) where F: FnMut() -> T {} } let b = X;
 b.iter(|| {
     let mut n = 1000_u32;
 
-    test::black_box(&mut n); // pretend to modify `n`
+    rustc_bench::black_box(&mut n); // pretend to modify `n`
 
     range(0, n).fold(0, |a, b| a ^ b)
 })

src/liballoc/heap.rs

@@ -370,8 +370,8 @@ mod imp {
 
 #[cfg(test)]
 mod test {
-    extern crate test;
-    use self::test::Bencher;
+    extern crate rustc_bench;
+    use self::rustc_bench::Bencher;
     use core::ptr::PtrExt;
     use heap;
 

src/libarena/lib.rs

@@ -507,8 +507,8 @@ impl<T> Drop for TypedArena<T> {
 
 #[cfg(test)]
 mod tests {
-    extern crate test;
-    use self::test::Bencher;
+    extern crate rustc_bench;
+    use self::rustc_bench::Bencher;
     use super::{Arena, TypedArena};
 
     #[allow(dead_code)]

src/libcollections/bench.rs

@@ -11,7 +11,7 @@
 use prelude::*;
 use std::rand;
 use std::rand::Rng;
-use test::{Bencher, black_box};
+use rustc_bench::{Bencher, black_box};
 
 pub fn insert_rand_n<M, I, R>(n: uint,
                               map: &mut M,

src/libcollections/bit.rs

@@ -2507,7 +2507,7 @@ mod bitv_bench {
     use std::rand;
     use std::rand::Rng;
     use std::u32;
-    use test::{Bencher, black_box};
+    use rustc_bench::{Bencher, black_box};
 
     use super::Bitv;
 
@@ -2526,7 +2526,7 @@ mod bitv_bench {
             for _ in range(0u, 100) {
                 bitv |= 1 << ((r.next_u32() as uint) % u32::BITS);
             }
-            black_box(&bitv)
+            black_box(&bitv);
         });
     }
 
@@ -2538,7 +2538,7 @@ mod bitv_bench {
             for _ in range(0u, 100) {
                 bitv.set((r.next_u32() as uint) % BENCH_BITS, true);
             }
-            black_box(&bitv)
+            black_box(&bitv);
         });
     }
 
@@ -3002,7 +3002,7 @@ mod bitv_set_bench {
     use std::rand;
     use std::rand::Rng;
     use std::u32;
-    use test::{Bencher, black_box};
+    use rustc_bench::{Bencher, black_box};
 
     use super::{Bitv, BitvSet};
 

src/libcollections/btree/map.rs

@@ -1614,7 +1614,7 @@ mod test {
 mod bench {
     use prelude::*;
     use std::rand::{weak_rng, Rng};
-    use test::{Bencher, black_box};
+    use rustc_bench::{Bencher, black_box};
 
     use super::BTreeMap;
     use bench::{insert_rand_n, insert_seq_n, find_rand_n, find_seq_n};

src/libcollections/dlist.rs

@@ -690,8 +690,7 @@ mod tests {
     use std::rand;
     use std::hash;
     use std::thread::Thread;
-    use test::Bencher;
-    use test;
+    use rustc_bench::Bencher;
 
     use super::{DList, Node};
 
@@ -1101,31 +1100,31 @@ mod tests {
     }
 
     #[bench]
-    fn bench_collect_into(b: &mut test::Bencher) {
+    fn bench_collect_into(b: &mut Bencher) {
         let v = &[0i; 64];
         b.iter(|| {
             let _: DList<int> = v.iter().map(|x| *x).collect();
         })
     }
 
     #[bench]
-    fn bench_push_front(b: &mut test::Bencher) {
+    fn bench_push_front(b: &mut Bencher) {
         let mut m: DList<int> = DList::new();
         b.iter(|| {
             m.push_front(0);
         })
     }
 
     #[bench]
-    fn bench_push_back(b: &mut test::Bencher) {
+    fn bench_push_back(b: &mut Bencher) {
         let mut m: DList<int> = DList::new();
         b.iter(|| {
             m.push_back(0);
         })
     }
 
     #[bench]
-    fn bench_push_back_pop_back(b: &mut test::Bencher) {
+    fn bench_push_back_pop_back(b: &mut Bencher) {
         let mut m: DList<int> = DList::new();
         b.iter(|| {
             m.push_back(0);
@@ -1134,7 +1133,7 @@ mod tests {
     }
 
     #[bench]
-    fn bench_push_front_pop_front(b: &mut test::Bencher) {
+    fn bench_push_front_pop_front(b: &mut Bencher) {
         let mut m: DList<int> = DList::new();
         b.iter(|| {
             m.push_front(0);
@@ -1143,31 +1142,31 @@ mod tests {
     }
 
     #[bench]
-    fn bench_iter(b: &mut test::Bencher) {
+    fn bench_iter(b: &mut Bencher) {
         let v = &[0i; 128];
         let m: DList<int> = v.iter().map(|&x|x).collect();
         b.iter(|| {
             assert!(m.iter().count() == 128);
         })
     }
     #[bench]
-    fn bench_iter_mut(b: &mut test::Bencher) {
+    fn bench_iter_mut(b: &mut Bencher) {
         let v = &[0i; 128];
         let mut m: DList<int> = v.iter().map(|&x|x).collect();
         b.iter(|| {
             assert!(m.iter_mut().count() == 128);
         })
     }
     #[bench]
-    fn bench_iter_rev(b: &mut test::Bencher) {
+    fn bench_iter_rev(b: &mut Bencher) {
         let v = &[0i; 128];
         let m: DList<int> = v.iter().map(|&x|x).collect();
         b.iter(|| {
             assert!(m.iter().rev().count() == 128);
         })
     }
     #[bench]
-    fn bench_iter_mut_rev(b: &mut test::Bencher) {
+    fn bench_iter_mut_rev(b: &mut Bencher) {
         let v = &[0i; 128];
         let mut m: DList<int> = v.iter().map(|&x|x).collect();
         b.iter(|| {

src/libcollections/lib.rs

@@ -33,7 +33,7 @@ extern crate core;
 extern crate unicode;
 extern crate alloc;
 
-#[cfg(test)] extern crate test;
+#[cfg(test)] extern crate rustc_bench;
 #[cfg(test)] #[macro_use] extern crate std;
 #[cfg(test)] #[macro_use] extern crate log;
 

src/libcollections/ring_buf.rs

@@ -1632,8 +1632,7 @@ mod tests {
     use core::iter;
     use std::fmt::Show;
     use std::hash;
-    use test::Bencher;
-    use test;
+    use rustc_bench::{Bencher, black_box};
 
     use super::RingBuf;
 
@@ -1750,15 +1749,15 @@ mod tests {
     }
 
     #[bench]
-    fn bench_new(b: &mut test::Bencher) {
+    fn bench_new(b: &mut Bencher) {
         b.iter(|| {
             let ring: RingBuf<u64> = RingBuf::new();
-            test::black_box(ring);
+            black_box(ring);
         })
     }
 
     #[bench]
-    fn bench_push_back_100(b: &mut test::Bencher) {
+    fn bench_push_back_100(b: &mut Bencher) {
         let mut deq = RingBuf::with_capacity(101);
         b.iter(|| {
             for i in range(0i, 100) {
@@ -1770,7 +1769,7 @@ mod tests {
     }
 
     #[bench]
-    fn bench_push_front_100(b: &mut test::Bencher) {
+    fn bench_push_front_100(b: &mut Bencher) {
         let mut deq = RingBuf::with_capacity(101);
         b.iter(|| {
             for i in range(0i, 100) {
@@ -1782,65 +1781,65 @@ mod tests {
     }
 
     #[bench]
-    fn bench_pop_back_100(b: &mut test::Bencher) {
+    fn bench_pop_back_100(b: &mut Bencher) {
         let mut deq: RingBuf<int> = RingBuf::with_capacity(101);
 
         b.iter(|| {
             deq.head = 100;
             deq.tail = 0;
             while !deq.is_empty() {
-                test::black_box(deq.pop_back());
+                black_box(deq.pop_back());
             }
         })
     }
 
     #[bench]
-    fn bench_pop_front_100(b: &mut test::Bencher) {
+    fn bench_pop_front_100(b: &mut Bencher) {
         let mut deq: RingBuf<int> = RingBuf::with_capacity(101);
 
         b.iter(|| {
             deq.head = 100;
             deq.tail = 0;
             while !deq.is_empty() {
-                test::black_box(deq.pop_front());
+                black_box(deq.pop_front());
             }
         })
     }
 
     #[bench]
-    fn bench_grow_1025(b: &mut test::Bencher) {
+    fn bench_grow_1025(b: &mut Bencher) {
         b.iter(|| {
             let mut deq = RingBuf::new();
             for i in range(0i, 1025) {
                 deq.push_front(i);
             }
-            test::black_box(deq);
+            black_box(deq);
         })
     }
 
     #[bench]
-    fn bench_iter_1000(b: &mut test::Bencher) {
+    fn bench_iter_1000(b: &mut Bencher) {
         let ring: RingBuf<int> = range(0i, 1000).collect();
 
         b.iter(|| {
             let mut sum = 0;
             for &i in ring.iter() {
                 sum += i;
             }
-            test::black_box(sum);
+            black_box(sum);
         })
     }
 
     #[bench]
-    fn bench_mut_iter_1000(b: &mut test::Bencher) {
+    fn bench_mut_iter_1000(b: &mut Bencher) {
         let mut ring: RingBuf<int> = range(0i, 1000).collect();
 
         b.iter(|| {
             let mut sum = 0;
             for i in ring.iter_mut() {
                 sum += *i;
             }
-            test::black_box(sum);
+            black_box(sum);
         })
     }
 

src/libcollections/slice.rs

@@ -2730,7 +2730,7 @@ mod bench {
     use core::ptr;
     use core::iter::repeat;
     use std::rand::{weak_rng, Rng};
-    use test::{Bencher, black_box};
+    use rustc_bench::{Bencher, black_box};
 
     #[bench]
     fn iterator(b: &mut Bencher) {