The study is relevant due to the opportunity of increasing efficiency of teaching mathematics at university through integration of students of computer experiment conducted with the use of IT in this process. The problem of there search is defined by a contradiction between great potential opportunities of mathematics experiment for motivating and developing modern scientific world outlook of students and beliefs about general methodology of cognition and between the lack of methodology for its implementation for teaching mathematics at university. The aim of the research is to explore methodological potential of computer experiment for improving the quality of teaching mathematics at university and to develop an appropriate methodology. The article describes student activities that allow using computer experiment for formation of individual research skills and skills of working with modern tools for solving theoretical and practical problems and presents methodology of studying certain sections of mathematical analysis including setting up problems that require experimental research of mathematical objects and their properties. The article may be used for content design of mathematics courses for raising motivation, preliminary studies of abstract notions, such as limit of a sequence and continuity, and organization of in-class, individual and research work of students aimed at exploring ways of scientific cognition and corresponding information technologies.